Configuring Multiple YARN Clusters

About this task

Multiple YARN clusters are created in the Mesos environment when multiple Myriad frameworks are each running a Resource Manager. The Resource Manager spawns Node Managers and Job History Service processes which form a cluster. To create multiple YARN clusters, a Resource Manager is run under the Myriad framework with the cluster name specified. The cluster name differentiates between staging, system, and local volume MapReduce directories. When configuring for multiple YARN clusters, run configure.sh on the nodes where you want the 1st Myriad framework and YARN cluster, then run configure.sh with different parameters on the nodes where you want the 2nd Myriad framework and YARN cluster, and so on.

Procedure

Run configure.sh on the slave nodes for the YARN cluster. Specify a Resource Manager hostname, Job History name, Myriad framework, and cluster prefix name.
NOTE: It is recommended that you use the same value for the -RM, -HS, -MF, and -MCL parameters.
For example, if you set the parameter value to framework1 for each of the parameters, the configure.sh parameters could be similar to the following:
```
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework1.marathon.mesos  
-HS jobhistory.framework1.mesos
-MF framework1 
-MCL framework1
                        
```
Run configure.sh on the slave nodes for each new YARN cluster. Specify new values for Resource Manager hostname, Job History name, Myriad framework, and cluster prefix name.
For example, on the subsequent YARN cluster, if you set the parameter value to framework2 for each of the parameters, the configure.sh parameters could be similar to the following:
```
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework2.marathon.mesos  
-HS jobhistory.framework2.mesos
-MF framework2 
-MCL framework2
                    
```

Example

For example, the following creates three Myriad frameworks and the associated YARN cluster name prefix (framework1, framework2, and framework3). In the following examples, -N is set to myCluster, the master IP addresses are 10.10.100.16, and the value for the -RM, -HS, -MF, and -MCL parameters changes for each YARN cluster.

Create the 1st cluster for framework1:

// On slave nodes 1 - 3, run:
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework1.marathon.mesos  
-HS jobhistory.framework1.mesos
-MF framework1 
-MCL framework1

Create the 2nd cluster for framework2:

// On slave nodes 4 - 6, run:
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework2.marathon.mesos  
-HS jobhistory.framework2.mesos
-MF framework2 
-MCL framework2

Create the 3rd cluster for framework3:

// On slave nodes 7 - 9, run:
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework3.marathon.mesos  
-HS jobhistory.framework3.mesos
-MF framework3 
-MCL framework3