mapred-site.xml
MapReduce is one type of application that can run on the Hadoop
2.x framework. MapReduce configuration options are stored in
the /opt/mapr/hadoop/hadoop-2.x.x/etc/hadoop/mapred-site.xml file
and are editable by the root user. This file contains configuration
information that overrides the default values for MapReduce
parameters. Overrides of the default values for core configuration
properties are stored in the Default mapred
Parameters file.
To override a default value for a property, specify the new
value within the <configuration> tags,
using the following format:
<property>
<name> </name>
<value> </value>
<description> </description>
</property>Configurations for MapReduce Applications
| Parameter | Value | Description |
|---|---|---|
| mapreduce.framework.name | yarn | Execution framework set to Hadoop YARN. |
| mapreduce.map.memory.mb | 1024 | Larger resource limit for maps. |
| mapreduce.map.java.opts | -Xmx1024M | Larger heap-size for child jvms of maps. |
| mapreduce.reduce.memory.mb | 3072 | Larger resource limit for reduces. |
| mapreduce.reduce.java.opts | -Xmx2560M | Larger heap-size for child jvms of reduces. |
| mapreduce.task.io.sort.mb | 512 | Higher memory limit while sorting data for efficiency. |
| mapreduce.task.io.sort.factor | 100 | More streams merged at once while sorting files. |
| mapreduce.reduce.shuffle.parallelcopies | 50 | Higher number of parallel copies run by reduces to fetch outputs from very large number of maps. |
Configurations for MapReduce JobHistory Server
| Parameter | Value | Description |
|---|---|---|
| mapreduce.jobhistory.address | MapReduce JobHistory Server host:port | Default port is 10020. |
| mapreduce.jobhistory.webapp.address | MapReduce JobHistory Server Web UI host:port | Default port is 19888. |
| mapreduce.jobhistory.intermediate-done-dir | /mr-history/tmp | Directory where history files are written by MapReduce jobs. |
| mapreduce.jobhistory.done-dir | /mr-history/done | Directory where history files are managed by the MapReduce JobHistory Server. |
Sample Hadoop 2.x mapred-site.xml File
The following mapred-site.xml file defines values for two job history
parameters.
<configuration>
<property>
<name>mapreduce.jobhistory.address</name>
<value>__HS_IP__:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>__HS_IP__:19888</value>
</property>
</configuration>