List of YARN Enhancements

MapR 5.2 runs the 2.7.0 version of Hadoop. In this MapR release, a number of bug fixes from various Hadoop versions were backported into 2.7.0.

The following bug fixes were backported from Hadoop 2.7.3 to Hadoop 2.7.0:
  • YARN-4452: NPE when submit Unmanaged application

  • YARN-4422: Generic AHS sometimes doesn't show started, node, or logs on App page

  • YARN-3849: Too much of preemption activity causing continuos killing of containers across queues

  • YARN-4794: Deadlock in NMClientImpl

  • YARN-4546: ResourceManager crash due to scheduling opportunity overflow

  • MAPREDUCE-6622: Add capability to set JHS job cache to a task-based limit

  • MAPREDUCE-6635: Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException

  • MAPREDUCE-6454: MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.

  • MAPREDUCE-6376: Add avro binary support for jhist files

  • MAPREDUCE-6554: MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory

  • MAPREDUCE-6302: Preempt reducers after a configurable timeout irrespective of headroom

  • MAPREDUCE-6514: Job hangs as ask is not updated after ramping down of all reducers

  • MAPREDUCE-6689: MapReduce job can infinitely increase number of reducer resource requests

  • MAPREDUCE-5485: Allow repeating job commit by extending OutputCommitter API

  • MAPREDUCE-6555: TestMRAppMaster fails on trunk

  • MAPREDUCE-6595: Fix findbugs warnings in OutputCommitter and FileOutputCommitter

  • MAPREDUCE-6478: Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob().

The following bug fixes were backported from Hadoop 2.7.2 to Hadoop 2.7.0:
  • YARN-4424: Fix deadlock in RMAppImpl

  • YARN-4354: Public resource localization fails with NPE

  • YARN-2902: Killing a container that is localizing can orphan resources in the DOWNLOADING state

  • YARN-4180: AMLauncher does not retry on failures when talking to NM

  • YARN-3697: FairScheduler: ContinuousSchedulingThread can fail to shutdown

  • YARN-3535: Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED

  • YARN-3857: Memory leak in ResourceManager with SIMPLE mode

  • YARN-3878: AsyncDispatcher can hang while stopping if it is configured for draining events on stop

  • YARN-3990: AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected

  • YARN-3793: Several NPEs when deleting local files on NM recovery

  • YARN-4047: ClientRMService getApplications has high scheduler lock contention

  • HADOOP-12304: Applications using FileContext fail with the default file system configured to be wasb/s3/etc.

  • MAPREDUCE-5948: org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well

  • MAPREDUCE-6481: LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes.

  • MAPREDUCE-6492: AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask

  • MAPREDUCE-5982: Task attempts that fail from the ASSIGNED state can disappear

  • MAPREDUCE-5649: Reduce cannot use more than 2G memory for the final merge

  • MAPREDUCE-6549: Multibyte delimiters with LineRecordReader cause duplicate records

The following bug fixes were backported from other Hadoop versions to Hadoop 2.7.0:
  • YARN-2336: Fair scheduler REST api returns a missing '[' bracket JSON for deep queue tree

  • YARN-1050: Document the Fair Scheduler REST API

  • YARN-3464: Race condition in LocalizerRunner kills localizer before localizing all resources

  • YARN-4041: Slow delegation token renewal can severely prolong RM recovery

  • YARN-4347: Resource manager fails with Null pointer exception

  • MAPREDUCE-6284: Add Task Attempt State API to MapReduce Application Master REST API

  • MAPREDUCE-6316: Task Attempt List entries should link to the task overview

  • MAPREDUCE-6364: Add a "Kill" link to Task Attempts pageAdd a "Kill" link to Task Attempts page

  • MAPREDUCE-6405: NullPointerException in App Attempts page

  • MAPREDUCE-6419: JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed 9999

  • MAPREDUCE-5649: Reduce cannot use more than 2G memory for the final merge