Tune the Reduce Phase for MapReduce v1

Configure the Percentage of Completed Map Tasks Before Reduce Tasks Can Start

In some cases, you can set the reducer to start before all map tasks are complete. The mapred.reduce.slowstart.completed.maps parameter controls the percentage of map tasks that must be complete before the reducers start. If the value of the mapred.reduce.slowstart.completed.maps parameter is set too low, random disk I/O results and performance will suffer.

If the output of the map tasks is large, set this to 0.95 to account for the overhead of starting the reducers. If the output of map tasks is small, you can lower this value. However, if you have a very large cluster or only one wave of map tasks, you may want to set this parameter to 0.

Configure the Number of Parallel Reduce Tasks

The reducer reads map results from every node, copying map outputs to its own buffers, sorting, and spilling when the buffer gets full. The mapred.reduce.parallel.copies parameter in the mapred-site.xml file controls how many map files the reducer can read in parallel. The default is 12. In most cases, it does not make sense to adjust this parameter. However, you can reduce it in very large clusters in order to throttle the network bandwidth used during the reduce phase if necessary.