Increasing Log Retention

To increase log retention for specific component, you can modify the configuration file and reset the value of the properties.

Increasing Log Retention for file system

In the /opt/mapr/conf/mfs.conf file, increase the value for the mfs.max.logfile.size.in.mb property. The value for this property is computed using the following formula:

maxSizePerLogFile = maxLogSize / MAX_NUM_OF_LOG_FILES
Here:
  • maxLogSize specifies the total amount of space that file system log files can consume.
  • MAX_NUM_OF_LOG_FILES specifies the total number of file system log files (5 hard coded).

For example, for a value of 10 GB, there will be 5 log files (mfs.log-3 to mfs.log-3.4) of 2GB each.

Increasing Log Retention for CLDB

NOTE This setting does not impact audit logging.

In the /opt/mapr/conf/log4j2.cldb.xml file, modify the SizeBasedTriggeringPolicy and DefaultRolloverStrategy properties.

  • Modify the size attribute for SizeBasedTriggeringPolicy that specifies the size of each cldb.log before rolling over to write to a new log file. For example, 
    <SizeBasedTriggeringPolicy size="1024MB"/>
  • Modify the max attribute for DefaultRolloverStrategy that specifies the number of cldb.log* files to be retained before the oldest one is deleted. For example,
    <DefaultRolloverStrategy max="20" fileIndex="min"/>

With the aforementioned values for SizeBasedTriggeringPolicy and DefaultRolloverStrategy, the CLDB log files can grow to 20GB (1024*20), before they are purged.

IMPORTANT You must restart CLDB on all nodes for the change in the values of CLDB log retention properties to take effect. Restart the passive CLDB nodes first, followed by restarting of the active CLDB node, so that the active CLDB node fails over to a passive CLDB node with the new values for CLDB log retention properties. Run the following command to restart a CLDB node. 
maprcli node services -cldb restart -nodes $(hostname)

Increasing Log Retention for Hadoop Services

You can increase log retention for ResourceManager, NodeManager, HistoryServer, and TimelineServer by modifying the following properties in the /opt/mapr/hadoop/hadoop-<Version>/etc/hadoop/log4j.properties file:

  • hadoop.log.maxfilesize — specifies the size of each <service-name>-hostname>.log before rolling over.
  • hadoop.log.maxbackupindex — specifies the number of <service-name>-<hostname>.log* before the oldest one is deleted.

For example, suppose the following configuration:

  • hadoop.log.maxfilesize = 1024Mb
  • hadoop.log.maxbackupindex = 10

There will be 10 GB of total service-specific logs (1GB per file) before the oldest file is purged.