MapR 5.1 is at End of Life (EOL) and no longer supported. Please see the latest documentation. This documentation is not being updated.

Home
5.1 Ecosystem
This guide contains a section for each open source project that MapR supports. You can learn how to install, configure, use, and integrate each project within the context of a MapR cluster.
Spark
Spark on YARN
Using Spark on YARN
This section includes information about using Spark on YARN in a MapR cluster.
Deployment Modes

MapR 5.1 Documentation

5.1 Ecosystem
This guide contains a section for each open source project that MapR supports. You can learn how to install, configure, use, and integrate each project within the context of a MapR cluster.
- AsyncHBase
- Cascading
- Drill
- Flume
- HBase
- HCatalog
- Hive
- HttpFS
- Hue
- Impala
- Mahout
- Myriad
- Oozie
- Pig
- OpenStack Sahara (Mitaka)
- Sentry
- Spark
  - Spark Feature Support
    MapR supports most Spark features. However, there a few exceptions.
  - Spark Standalone
  - Spark on YARN
    - Installing Spark on YARN
    - Upgrading Spark on YARN
    - Configuring Spark on YARN
    - Using Spark on YARN
      This section includes information about using Spark on YARN in a MapR cluster.
      - Deployment Modes
      - Run Spark from the Spark Shell
  - Integrating Spark
  - Spark 2.0.0 Developer Preview
    Apache Spark is an open-source processing engine that you can use to process Hadoop data. Although MapR does not yet ship a Spark 2.0.0 package, you can install and use Spark 2.0.0 on a non-secure MapR 5.1 cluster or on a secure MapR 5.1 cluster that uses MapR-SASL authentication.
- Sqoop
- Storm
- Third Party Solutions

Deployment Modes

Spark is preconfigured for YARN and does not require any additional configuration to run.

Two deployment modes can be used to launch Spark applications on YARN:

In yarn-cluster mode, jobs are managed by the YARN cluster. The Spark driver runs inside an Application Master (AM) process that is managed by YARN. This means that the client can go away after initiating the application.
In yarn-client mode, the Spark driver runs in the client process, and the Application Master is only used to request resources from YARN.

MapR recommends using yarn-cluster mode instead of yarn-client mode. If the Spark client that runs the job exits after submitting the job, there is no impact on actual job completion.

Note: In yarn-cluster mode, the local directories used by the Spark executors and the Spark driver are the local directories that are configured for YARN (yarn.nodemanager.local-dirs). If you specify a different path with SPARK_LOCAL_DIRS (as you would for Spark running in standalone mode), that path will be ignored.