Hadoop Client Compatibility

Describes compatibility between Hadoop 2.x and Hadoop 3.x clients and servers.

Hadoop Client-Server Compatibility Matrix

As indicated in Hadoop Protocol Versions, EEP 9.0.0 and later include Hadoop 3, while previous EEP versions used Hadoop 2. The following table summarizes Hadoop 2 and Hadoop 3 client-server compatibility:
Hadoop 3 Server Hadoop 2 Server
Hadoop 3 Client Compatible Not Compatible
Hadoop 2 Client Compatible1 Compatible
1Some configuration is required. Use the following step.

Using a Hadoop 2 Client with a Hadoop 3 Server

To use a Hadoop 2 client with a Hadoop 3 server, add the following properties to the <HADOOP_HOME>/etc/hadoop/mapred-site.xml file:
<property>
  <name>yarn.app.mapreduce.am.staging-dir</name>
  <value>/var/mapr/cluster/yarn/rm/staging</value>
</property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory on the server node}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory on the server node}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory on the server node}</value>
</property>

The first property change is needed for the Hadoop 2 client contained in EEP 8.1.0. In EEP 8.1.0, the default path was changed to /var/mapr/cluster/yarn/hs. In EEP 9.0.0 and later, the default was changed back to /var/mapr/cluster/yarn/rm/staging.