Integrate Spark with HBase
Integrate Spark with HBase or MapR-DB when you want to run Spark jobs on HBase or MapR-DB tables.
About this task
Procedure
-
Configure the HBase version in the
/opt/mapr/spark/spark-<version>/mapr-util/compatibility.version
file:hbase_versions=<version>
-
If you want to create HBase tables with Spark, add following property to
hbase-site.xml:
<property> <name>hbase.table.sanity.checks</name> <value>false</value> </property>
-
Copy the hbase-site.xml to
{SPARK_HOME}/conf/
directory on each Spark node. - For Spark 1.4.1 or Spark 1.5.2-1512: Add the following line to spark.executor.extraClassPath in the /opt/mapr/spark/spark-<version>/ conf/spark-defaults.conf file: /opt/mapr/hbase/hbase-<version>/lib/*
-
To verify the integration, complete the following steps: