Enabling YARN Log Aggregation
To enable YARN log aggregation, add or edit the following
properties in yarn-site.xml
:
- Set the value of
yarn.log-aggregation-enable
totrue.
- Optional: Set the value of
yarn.nodemanager.remote-app-log-dir
to a location in the MapR-FS. By default, the location ismaprfs:///tmp/logs
. - Optional: Set the value of
yarn.nodemanager.remote-app-log-dir-suffix
to the name of the folder that should contain the logs for each user. By default, the folder name islogs
.
On a non-secure cluster, you also need to add the following
property to /opt/mapr/hadoop/hadoop-2.x/etc/hadoop/yarn-env.sh
on the Node Manager nodes:
export MAPR_IMPERSONATION_ENABLED=1
Then restart the Node Manager services. This setting enables impersonation for Node Manager processes so that log files can be created with the correct user ownership.
Aggregated logs are owned by the user who runs the job. For example, when user admin runs a
job, the logs are stored to maprfs:///tmp/logs/admin
. If user analyst runs
a job, the logs are stored to maprfs:///tmp/logs/analyst
. If these two
users do not share the same UNIX group, they will not be able to see each other's logs.