Allocating Cluster Resource from the Command-Line

Provides a general overview on allocating cluster resources for a data-fabric Hadoop cluster.

In a data-fabric Hadoop cluster, the warden sets the default resource allocation for the operating system, file system, data-fabric Hadoop services, and YARN applications. Warden allocates resources to data-fabric Hadoop services and applications based on the roles installed on a node. For example, warden allocates resources for YARN applications on nodes with NodeManager role installed.

In general, you should not need to override the values set in the default configuration files and by warden. However, you can provide updated values by adding or updating parameters in the Hadoop site configuration files or warden files. To override parameter values for a single job, the option can be overridden in the command line when submitting a YARN application to the cluster.

To determine the current value of a hadoop parameter, run hadoop conf | grep <ParameterName>. In the following example, the hadoop conf command was used to get the value of mapreduce.map.memory.md:

hadoop conf | grep mapreduce.map.memory.mb
<property><name>mapreduce.map.memory.mb</name><value>1024</value><source>mapred-site.xml</source></property>

Alternatively, run hadoop conf without the grep command to get a full list of the current parameter values. To determine the current value of a warden parameter, open the warden files located in the following directories: /opt/mapr/conf/conf.d and /opt/mapr/conf.

NOTE In some cases, the current value of the parameter can only be seen in the Control System or in ResourceManager.

Refer to Allocating Memory for Nodes to allocate memory and resources in a data-fabric cluster.