Configuring Kerberos User Authentication

Configuring Kerberos for Authentication Using MapR Tickets

To use Kerberos to generate MapR tickets for users, enable Kerberos on the CLDB by creating a Kerberos identity on the Kerberos server used by the cluster and distributing that identity to the other CLDB nodes in the cluster.

MapR clusters do not provide Kerberos infrastructure. This section assumes you have a functioning Kerberos realm and your systems have the Kerberos client installed. The tips in this section assume a Linux-based Kerberos environment, and the specific commands for your environment may vary. Please consult with your Kerberos administrator for assistance.

Creating a Kerberos Identity for the CLDB

The CLDB requires a Kerberos server identity, but no other nodes do. By default, this identity takes the form mapr/<cluster name>. You can use configure.sh or edit the mapr-clusters.conf file to change this default. Use the following commands in a Linux-based Kerberos environment to set up the identity:

kadmin
        : addprinc -randkey mapr/my.cluster.com
        : ktadd -k /opt/mapr/conf/mapr.keytab mapr/my.cluster.com

Copy the resulting mapr.keytab file to the same location on every CLDB node. The mapr.keytab file must be owned and readable only by the mapr user. You can specify the location of the mapr.keytab file in the conf/mapr.login.conf file. The default location for mapr.keytab is /opt/mapr/conf.

Updating the keytab File

You can use the kadmin tool to update the server keys that are stored in the keytab file. Because the server tickets used to authenticate to the CLDB use the new keys immediately, you must copy the new keytab file to all the CLDB servers in the cluster immediately after updating the server keys.

To update the keytab file with a new key, run the following command:

kadmin
        : ktadd -k /opt/mapr/conf/mapr.keytab mapr/my.cluster.com

The CLDB automatically detects changes to the keytab file on systems that use Java 7 or later. Systems that use Java 6 require a CLDB restart to detect changes to the keytab file.

Running configure.sh

After a Kerberos principal is created for the CLDB, that principal is added to the mapr.keytab file, and the mapr.keytab file is copied to all the CLDB servers, Kerberos user authentication is fully enabled for the MapR cluster.

Two configure.sh parameters are important for Kerberos:

• -K|-kerberosEnable - lets the rest of the cluster know that Kerberos is enabled, so that clients can auto detect Kerberos tickets and use them to get MapR tickets.
• -P "<cldbPrincipal>"- specifies the Kerberos instance which is used to form the CLDB Kerberos principal in the form of mapr/<instance-name>@<realm-name>. Enclose this value in quotes (").

configure.sh -K -P "<cldbPrincipal>" 

Running configure.sh on each node enters the Kerberos information into the local clusters.conf file, so that the following command is all that is required for the client to access the cluster:

hadoop fs -ls

maprlogin kerberos
hadoop fs -ls

Kerberos Command Summary

• kinit: Creates a Kerberos ticket. Prompts the user for userid and password. After validating, Kerberos creates a ticket file in /tmp that is owned by the user. Use the -R option to renew an existing ticket. Kerberos credentials expire in 8-10 hours. Expired credentials must be renewed or replaced. By default, tickets can be renewed for up to 24 hours.
• klist: Lists the contents of the user's ticket file.
• kdestroy: Destroys the contents of the user's ticket file. The user is no longer authenticated.
• kadmin: Used to administer Kerberos. The login for this command is implicitly <userid>/admin, since administrator ids typically end in /admin.
• ktutil: Kerberos keytab maintenance utility. Used to combine, or alter Kerberos keytabs.

Kerberos Troubleshooting

Java errors from Kerberos problems can be obscure and difficult to interpret. To see the Kerberos error messages, enable Kerberos debugging by adding these settings to your JVM:

-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true -Djavax.net.debug=all

You can also enable Kerberos debugging for the MapR-provided maprcli and Hadoop clients by adding the following line to the env.sh shell script:

#MAPRLOGIN_OPTS="${MAPRLOGIN_OPTS} -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true -Djavax.net.debug=all"

Capture the Kerberos error to research the issue.

The following sections list common Kerberos error conditions:

Incorrect JVM

Nodes often have multiple Java Virtual Machines (JVM) installed. The MapR env.sh script automatically configures a JVM to use. To change the automatically configured JVM, set the value of the JAVA_HOME environment variable in the /opt/mapr/conf/env.sh file.

Incorrect Server Name

The following error message is caused by an incorrect CLDB server name in the mapr.login.conf file. The error message mentions passwords, but the error condition is unrelated to password authentication.

2013-04-25 16:46:02,324 ERROR MapRLoginServlet [185087767@qtp-648535353-2]: Failed to obtain kerberos identity, continuing anyway...
javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:789)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:654)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
        at com.mapr.fs.cldb.http.login.MapRLoginServlet.init(MapRLoginServlet.java:73)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:339)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Invalid or missing keytab file

The keytab file must be consistent with the key versions of the Kerberos principal. The following example shows an inconsistent keytab file:

kadmin:  getprinc mapr/realm1
Principal: mapr/realm1@mapr
Expiration date: [never]
Last password change: Thu May 23 15:36:01 PDT 2013
Password expiration date: [none]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Thu May 23 15:36:01 PDT 2013 (mapr/admin@mapr)
Last successful authentication: Thu May 23 19:31:59 PDT 2013
Last failed authentication: Thu May 23 15:35:41 PDT 2013
Failed password attempts: 0
Number of keys: 8
Key: vno 15, aes256-cts-hmac-sha1-96, no salt
Key: vno 15, arcfour-hmac, no salt
Key: vno 15, des3-cbc-sha1, no salt
Key: vno 15, des-cbc-crc, no salt
Key: vno 15, des-cbc-md5, Version 4
Key: vno 15, des-cbc-md5, Version 5 - No Realm
Key: vno 15, des-cbc-md5, Version 5 - Realm Only
Key: vno 15, des-cbc-md5, AFS version 3
MKey: vno 1

ktutil:  rkt mapr.keytab
ktutil:  l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1   14             mapr/realm1@mapr
   2   14             mapr/realm1@mapr
   3   14             mapr/realm1@mapr
   4   14             mapr/realm1@mapr
ktutil:  q

Note that the key versions in the Kerberos principal /realm1 are 15, and the versions in the keytab file are 14. This mismatch can result in errors about missing keys or mismatched encryption.

$ chown mapr:mapr /opt/mapr/conf/mapr.keytab

Incompatible encryption on Java runtime

Incompatible cryptography between the KDC and the JDK results in failed handshakes, leading to errors similar to the following:

Caused by: javax.security.auth.login.LoginException: Unable to obtain Principal Name for authentication 

With debugging active, the following message is displayed:

>>>DEBUG <CCacheInputStream>  client principal is username@hostname
>>>DEBUG <CCacheInputStream> server principal is X-CACHECONF:/krb5_ccache_conf_data/fast_avail/krbtgt/user@hostname
>>>DEBUG <CCacheInputStream> key type: 0
>>>DEBUG <CCacheInputStream> auth time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG <CCacheInputStream> start time: null
>>>DEBUG <CCacheInputStream> end time: Wed Dec 31 16:00:00 PST 1969
>>>DEBUG <CCacheInputStream> renew_till time: null
>>> CCacheInputStream: readFlags() 
>>> unsupported key type found the default TGT: 18
Negotiate support not initiated, will fallback to other scheme if allowed. Reason:

This debug message indicates that the problem is an unsupported key type.

Incompatible encryption errors can occur due to a keytab file that is not present or contains outdated keys.

Be sure to update the Java jurisdiction policy file. Jurisdiction policy files are available from Oracle.

A persistent encryption incompatibility problem may require you to edit the krb5.conf file to ensure compatible algorithms between Java and Kerberos.

Bugs in Java

The following error occurs in Java version 1.6.0_25. Upgrade to 1.6.0_45 to resolve the error.

java.io.IOException: extra data given to DerValue constructor
        at sun.security.util.DerValue.init(DerValue.java:368)
        at sun.security.util.DerValue.<init>(DerValue.java:277)
        at sun.security.krb5.internal.Ticket.<init>(Ticket.java:81)
        at sun.security.krb5.internal.ccache.CCacheInputStream.readData(CCacheInputStream.java:250)
        at sun.security.krb5.internal.ccache.CCacheInputStream.readCred(CCacheInputStream.java:357)
        at sun.security.krb5.internal.ccache.FileCredentialsCache.load(FileCredentialsCache.java:225)
        at sun.security.krb5.internal.ccache.FileCredentialsCache.acquireInstance(FileCredentialsCache.java:104)
        at sun.security.krb5.internal.ccache.CredentialsCache.getInstance(CredentialsCache.java:75)
        at sun.security.krb5.Credentials.acquireTGTFromCache(Credentials.java:304)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:589)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$5.run(LoginContext.java:706)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:703)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:575)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:554)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:528)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1656)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1649)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1517)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:235)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
        at org.apache.hadoop.fs.FsShell.init(FsShell.java:87)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:1808)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1967)
13/05/10 15:24:00 DEBUG security.SaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos principal name is hdfs/qa-node24@dev-maprtech
13/05/10 15:24:00 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

Kerberos and PAM validation

Standard Kerberos implementations are predicated on access to elevated user privileges that are not present on secure MapR clusters. In a MapR cluster, the MCS console and other components call PAM as an ordinary user process. This discrepancy in expected and actual privileges can cause a variety of obscure file permisson errors. Since different Kerberos PAM modules are available, error reports can vary.

To diagnose this issue, attempt starting the MCS as the root user, or clear out the /tmp folder. If there are no problems when starting MCS as root, or if clearing out the /tmp folder enables a single login before errors appear again, the problem may lie in the Kerberos PAM configuration.

To resolve this condition, prevent Kerberos from creating a ticket file. MapR security does not use Kerberos tickets. The Kerberos KDC is used to validate passwords. Typically the configuration file for PAM is in the /etc/pam.d directory. See the documentation for your specific Kerberos PAM module for more information.

Disabling Replay Detection for Kerberos Authentication

You can set an option in mapr-clusters.conf to disable replay detection for Kerberos runtime authentication.

disableReplayDetection=true

By default, this parameter is set to false, meaning that MapR clients enable Kerberos replay detection. Typically, replay detection is enabled to prevent potential attacks such as the replay of Kerberos packets or multiple login attempts with the same user ID. Set this parameter to true only if you want MapR clients not to enforce this detection.

This parameter applies when users attempt an implicit or explicit maprlogin, such as by using the maprlogin kerberos command or by submitting jobs and other operations with kerberosEnable=true set in the mapr-clusters.conf file.

This parameter is used when applications connect to the cluster using Kerberos; mapr-clusters.conf only needs to be updated when it is used by such applications. If all Kerberos access to the cluster is from clients outside the cluster, only the mapr-clusters.conf file on those client machines has to be updated. If Kerberos is used from applications running on the cluster, mapr-clusters.conf should be updated there as well.