MapR 5.1 is at End of Life (EOL) and no longer supported. Please see the latest documentation. This documentation is not being updated.

Home
5.1 Ecosystem
This guide contains a section for each open source project that MapR supports. You can learn how to install, configure, use, and integrate each project within the context of a MapR cluster.
Spark
Spark on YARN
Configuring Spark on YARN
Security for Spark on YARN
Configure SSL Encryption for Spark on YARN
As of Spark 1.5.2-1605 and Spark 1.6.1-1605, encryption between all the SparkMaster and SparkWorker nodes is disabled by default. In Spark 1.5.2-1603 and 1.6.1-1604, when the cluster is secure, encryption between all of the SparkMaster and SparkWorker nodes is enabled by default using the TLS v1.2 protocol. Complete the following step to manually configure encryption.

MapR 5.1 Documentation

5.1 Ecosystem
This guide contains a section for each open source project that MapR supports. You can learn how to install, configure, use, and integrate each project within the context of a MapR cluster.
- AsyncHBase
- Cascading
- Drill
- Flume
- HBase
- HCatalog
- Hive
- HttpFS
- Hue
- Impala
- Mahout
- Myriad
- Oozie
- Pig
- OpenStack Sahara (Mitaka)
- Sentry
- Spark
  - Spark Feature Support
    MapR supports most Spark features. However, there a few exceptions.
  - Spark Standalone
  - Spark on YARN
    - Installing Spark on YARN
    - Upgrading Spark on YARN
    - Configuring Spark on YARN
      - Configure MapR Client Node to Run Spark Applications
        When Spark runs on YARN, MapR client nodes require the hadoop-yarn-server-web-proxy JAR file to run Spark applications. A MapR client node (a node with the mapr-client package, but without mapr-core packages) is also known as an edge node.
      - Configure Spark JAR Location
      - Configure Spark with the NodeManager Local Directory Set to MapR-FS
      - Security for Spark on YARN
        Configure Authentication for Spark on YARN
        As of Spark 1.5.2-1603 and Spark 1.6.1-1604, when the cluster is secure, authentication via a shared secret is enabled by default between all of the SparkMaster and SparkWorker nodes. When authentication is enabled, authentication keys are randomly generated for each job. When the cluster is not secure or for previous versions of Spark, complete the following step to manually configure authentication.
        Configure SSL Encryption for Spark on YARN
        As of Spark 1.5.2-1605 and Spark 1.6.1-1605, encryption between all the SparkMaster and SparkWorker nodes is disabled by default. In Spark 1.5.2-1603 and 1.6.1-1604, when the cluster is secure, encryption between all of the SparkMaster and SparkWorker nodes is enabled by default using the TLS v1.2 protocol. Complete the following step to manually configure encryption.
        Configure SASL Encryption for Spark on YARN
        As of Spark 1.6.1-1605, when the cluster is secure, SASL encryption between all the SparkMaster and SparkWorker nodes is enabled by default. When the cluster is not secure or for previous versions of Spark, complete the following steps to manually enable SASL encryption.
    - Using Spark on YARN
      This section includes information about using Spark on YARN in a MapR cluster.
  - Integrating Spark
  - Spark 2.0.0 Developer Preview
    Apache Spark is an open-source processing engine that you can use to process Hadoop data. Although MapR does not yet ship a Spark 2.0.0 package, you can install and use Spark 2.0.0 on a non-secure MapR 5.1 cluster or on a secure MapR 5.1 cluster that uses MapR-SASL authentication.
- Sqoop
- Storm
- Third Party Solutions

Configure SSL Encryption for Spark on YARN

As of Spark 1.5.2-1605 and Spark 1.6.1-1605, encryption between all the SparkMaster and SparkWorker nodes is disabled by default. In Spark 1.5.2-1603 and 1.6.1-1604, when the cluster is secure, encryption between all of the SparkMaster and SparkWorker nodes is enabled by default using the TLS v1.2 protocol. Complete the following step to manually configure encryption.

About this task

Procedure

In the spark-defaults.conf on each spark node, configure the following properties:

spark.ssl.akka.enabled true
spark.ssl.fs.enabled true
spark.ssl.keyPassword mapr123
spark.ssl.keyStore /opt/mapr/conf/ssl_keystore
spark.ssl.keyStorePassword mapr123
spark.ssl.trustStore /opt/mapr/conf/ssl_truststore
spark.ssl.trustStorePassword mapr123
spark.ssl.protocol TLSv1.2
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA

The spark-defaults.conf file is in the following location: /opt/mapr/spark/spark-<version>/conf/

NOTE: When you manually configure encryption between the SparkMaster and SparkWorker nodes, configure the same protocol and algorithms for each node. Otherwise, the connection between those components will fail.