ExpressLane
MapR provides an express path (called ExpressLane) that works in conjunction with the Fair Scheduler. ExpressLane is for small MapReduce jobs to run when all slots are occupied by long tasks. ExpressLane is not available on YARN.
Small jobs are only given this special treatment when the cluster is busy, and only if they
meet the criteria specified by the following parameters in mapred-site.xml:
| Parameter | Value | Description |
|---|---|---|
| mapred.fairscheduler.smalljob.schedule.enable | true | Enable small job fast scheduling inside fair scheduler. TaskTrackers should reserve a slot called ephemeral slot, which is used for smalljob if the cluster is busy. |
| mapred.fairscheduler.smalljob.max.reducers | 10 | Small job definition. Max number of reducers allowed in small job. |
| mapred.fairscheduler.smalljob.max.reducer.inputsize | 1073741824 | Small job definition. Max estimated input size for a reducer allowed in small job. Default is 1GB per reducer. |
| mapred.fairscheduler.smalljob.max.maps | 10 | Small job definition. Max number of maps allowed in small job. |
| mapred.fairscheduler.smalljob.max.inputsize | 10737418240 | Small job definition. Max input size in bytes allowed for a small job. Default is 10GB. |
| mapred.cluster.ephemeral.tasks.memory.limit.mb | 200 | Small job definition. Max memory in mbytes reserved for an ephermal slot. Default is 200mb. This value must be same on JobTracker and TaskTracker nodes. |
MapReduce jobs that appear to fit the small job definition but are in fact larger than anticipated are killed and re-queued for normal execution.