MapR 5.1 is at End of Life (EOL) and no longer supported. Please see the latest documentation. This documentation is not being updated.

Home
5.1 Ecosystem
This guide contains a section for each open source project that MapR supports. You can learn how to install, configure, use, and integrate each project within the context of a MapR cluster.
Spark
Integrating Spark
Integrate Spark with MapR Streams
Integrate Spark with MapR Steams to enable Spark to query MapR Streams for new messages at a given interval, process any new messages that are available, and also publish messages into MapR Streams.

MapR 5.1 Documentation

5.1 Ecosystem
This guide contains a section for each open source project that MapR supports. You can learn how to install, configure, use, and integrate each project within the context of a MapR cluster.
- AsyncHBase
- Cascading
- Drill
- Flume
- HBase
- HCatalog
- Hive
- HttpFS
- Hue
- Impala
- Mahout
- Myriad
- Oozie
- Pig
- OpenStack Sahara (Mitaka)
- Sentry
- Spark
  - Spark Feature Support
    MapR supports most Spark features. However, there a few exceptions.
  - Spark Standalone
  - Spark on YARN
  - Integrating Spark
    - Integrate Spark-SQL with Avro
      Integrate Spark-SQL with Avro when you want to read and write Avro data.
    - Integrate Spark with HBase
      Integrate Spark with HBase or MapR-DB when you want to run Spark jobs on HBase or MapR-DB tables.
    - Integrate Spark-SQL with Hive
      Integrate Spark-SQL with Hive when you want to run Spark-SQL queries on Hive tables.
    - Integrate Spark with MapR Streams
      Integrate Spark with MapR Steams to enable Spark to query MapR Streams for new messages at a given interval, process any new messages that are available, and also publish messages into MapR Streams.
      - Configure Spark to Consume MapR Streams Messages
        Using the Kafka 0.9 API, you can configure a Spark application to query MapR Streams for new messages at a given interval.
      - Configure Spark to Produce MapR Streams Messages
        Using the Kafka 0.9 API, you can configure a Spark application to produce MapR Streams messages.
    - Integrate Spark with R
      Integrate Spark with R when you want to run R programs as Spark jobs.
  - Spark 2.0.0 Developer Preview
    Apache Spark is an open-source processing engine that you can use to process Hadoop data. Although MapR does not yet ship a Spark 2.0.0 package, you can install and use Spark 2.0.0 on a non-secure MapR 5.1 cluster or on a secure MapR 5.1 cluster that uses MapR-SASL authentication.
- Sqoop
- Storm
- Third Party Solutions

Integrate Spark with MapR Streams

Integrate Spark with MapR Steams to enable Spark to query MapR Streams for new messages at a given interval, process any new messages that are available, and also publish messages into MapR Streams.

As of Spark 1.5.2-1602, you can use Spark to access MapR Streams through Spark’s receiver-less, direct approach. The integration between Spark 1.5.2 and MapR Streams is a beta feature.

For more information, see the Apache Spark documentation.

NOTE: Before you integrate Spark with MapR Streams, verify that the Streams Client is install on all Spark nodes. For more information, see Installing Spark on YARN or Installing Spark Standalone.