Example Map and Reduce Slot Calculation

In the following example, the node has the following configuration:

Node Resources or Settings Values
Services and Options TaskTracker, MapR-FS, MapR-DB
CPU/Core 24
Disks Available to MapR-FS 5
RAM 48G
Chunk Size 256MB

Based on this configuration, MapR Hadoop performs the following calculations to determine the number of map and reduce slots:

Calculation Value Description
Number of CPUs 4 Since MapR-DB is running, 4 CPUs are used to determine the slot calculation.
Memory for Map Slots 1G

Since the chunk size is 256, 1G is allocated to memory for map slots. Warden sets mapred.job.map.memory.physical.mb to 1000MB.

Memory for Reduce Slots

 3G

Since the chunk size is 256, 3G is allocated to memory for map slots. Warden sets mapred.job.reduce.memory.physical.mb to 3000MB.
Memory available to process MapReduce V1 tasks 26G

Based on the services running on the node, Warden calculates the memory available to process MapReduce v1 tasks. In this example, Warden sets mapreduce.tasktracker.reserved.physicalmemory.mb to 26000MB. For more information, see Memory Allocation for Nodes.
Map Slots 10

This value is based on the following calculations:

= Min [ (0.4 * mapreduce.tasktracker.reserved.physicalmemory.mb)/ mapred.job.map.memory.physical.mb, (CPU > 2 ? 2 * (CPU - MapRFSCPU) : 1), 2 * MapRFSdisks = Min[ (0.4*26)/1, 2* 23-4), 2*5] = Min [ 10.4, 38, 10] = 10

In this example, Warden sets mapred.tasktracker.map.tasks.maximum to 10.
Reduce Slots 3

This value is based on the following calculation: = Min [(0.6 * mapreduce.tasktracker.reserved.physicalmemory.mb)/ mapred.job.reduce.memory.physical.mb, (CPU > 2 ? CPU - MapRFSCPU : 1), (MapRFSDISKS > 2 ? 0.75 * MapRFSDISKS : 1) ] = Min[ (0.6*26)/3, 23-4, 0.75*5] = Min [5.2, 19, 3.75] = 3 In this example, Warden sets mapred.tasktracker.reduce.tasks.maximum to 3.

Customizing the MapReduce v1 Slot Calculation Parameters

In general, you should not need to customize the number of map and reduce slots because Warden determines these value based on the resource available on the node. However, you can override the number of slots by adding one or more of these parameters to mapred-site.xml. The mapred-site.xml file for MapReduce v1 jobs is in the following location: /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml.

NOTE: If you make changes to mapred-site.xml, you must restart TaskTracker. Warden uses the following parameters to calculate and assign values to map slots and reduce slots on each node:
Parameter Default Value Description

mapreduce.tasktracker.reserved.physicalmemory.mb

Warden uses the following formula to calculate this value: [total physical memory on node] – [memory required by the operating system, MapR file system and MapR services installed on the node]-[memory allocated to YARN applications, if Node Manager is installed on the node]. For more information, see Memory Allocation for Nodes. To determine the value, go to the TaskTracker UI and view the memory available for that node.

 Defines the memory available to process MapReduce v1 tasks in MB.
mapred.tasktracker.map.tasks.maximum Warden uses a formula to calculate this value. For more information, see Criteria for Map Slot Calculation. Defines the maximum number of MapReduce v1 map slots.
mapred.tasktracker.reduce.tasks.maximum Warden uses a formula to calculate this value. For more information, see Criteria for Reduce Slot Calculation. Defines the maximum number of MapReduce v1 map slots.
mapred.job.map.memory.physical.mb

If the chunk size is greater than or equal to 256M, then this value is set to 1G. Otherwise, this value is set to 0.5G. For information about configuring the chunk size, see Chunk Size.

Defines the amount of memory allocated to map tasks in MB.
mapred.job.reduce.memory.physical.mb

If the chunk size is greater than or equal to 256M, then this value is set to 3G. Otherwise, this value is set to 1.5G. For information about configuring the chunk size, see Chunk Size.

 Defines the amount of memory allocated to reduce tasks in MB.