Configuring Multiple YARN Clusters (Binary)

This topic describes how to set up multiple Myriad frameworks and multiple YARN clusters in the Mesos environment for binary distribution.

About this task

Procedure

On all Mesos slave nodes, create a new default directory for Mesos slave information.
1. Change directory to /etc/mesos-slave.
```
cd /etc/mesos-slave/
```
2. Create a new directory path and a file called work_dir and add /opt/mapr as the new Mesos slave working directory.
```
echo "new directory" > work_dir
vi work_dir
/opt/mapr
                            
```
3. Stop all Mesos slaves using the maprcli node services command.
```
maprcli node services -name mesos-slave -action stop -nodes <Nodes where Mesos slaves are running>
```
4. Restart all Mesos slaves using the maprcli node services command.
```
maprcli node services -name mesos-slave -action restart -nodes <Nodes where Mesos slaves are running>
```
  NOTE: If your Mesos slaves do not restart, remove the original directories used by the Mesos slave nodes. Typically, /tmp/mesos/meta/slaves/latest
```
rm /tmp/mesos/meta/slaves/latest
```

Set up a Myriad framework.

Run configure.sh to specify a Resource Manager hostname, Myriad framework, Job History name, and cluster prefix name.
The following shows example parameters, if framework1 is the value for the configure.sh -RM, -HS, -MF, and -MCL parameters.
```
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework1.marathon.mesos  
-HS jobhistory.framework1.mesos
-MF framework1 
-MCL framework1
                        
```

Copy the myriad-config-default.yml file from /opt/mapr/myriad/myriad-0.1/conf/ to /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/.


cp /opt/mapr/myriad/myriad-0.1/conf/myriad-config-default.yml /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/

Edit the myriad-config-default.yml file in the /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/ and update the nodeManagerUri and YARN_HOME parameters:


nodeManagerUri: maprfs://<full path to Hadoop tar file>
    // For example: maprfs:///dist/hadoop-2.7.0.framework1.tar.gz
YARN_HOME: hadoop-2.7.0            
    //  (or the relative path to the hadoop directory that will be created in the tar file)

Create a tar file for the Myriad framework.


cd /opt/mapr/hadoop
tar -czvf hadoop-2.7.0.framework1.tar.gz hadoop-2.7.0

Copy the Myriad framework tar file to the location specified by the nodeManagerUri parameter in the myriad-config-default.yml file.
Repeat steps 2, 3, and 4 for each Myriad framework.

Example

For example, the following creates two Myriad frameworks and the associated YARN cluster name prefix (framework1 and framework2). In the following examples, the value is the same for the configure.sh -RM. -HS, -MF, and -MCL parameters.

Create the 1st tar for framework1:

// Step 2a
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework1.marathon.mesos  
-HS jobhistory.framework1.mesos
-MF framework1 
-MCL framework1

// Step 2b
cp /opt/mapr/myriad/myriad-0.1/conf/myriad-config-default.yml /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/

// Step 2c
vi /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/myriad-config-default.yml

nodeManagerUri: maprfs:///dist/hadoop-2.7.0.framework1.tar.gz
YARN_HOME: hadoop-2.7.0

// Step 3
cd /opt/mapr/hadoop/
tar -czvf hadoop-2.7.0.framework1.tar.gz hadoop-2.7.0

// Step 4
cp hadoop-2.7.0.framework1.tar.gz maprfs:///dist/hadoop-2.7.0.framework1.tar.gz

Create the 2nd tar for framework2:

// Step 2a
/opt/mapr/server/configure.sh 
-C 10.10.100.16
-Z 10.10.100.16 -u mapr -g mapr 
-N myCluster 
-F /root/disk.list 
-RM framework2.marathon.mesos  
-HS jobhistory.framework2.mesos
-MF framework2 
-MCL framework2

// Step 2b                    
cp /opt/mapr/myriad/myriad-0.1/conf/myriad-config-default.yml /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/

// Step 2c
vi /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/myriad-config-default.yml
                    
nodeManagerUri: maprfs:///dist/hadoop-2.7.0.framework2.tar.gz
YARN_HOME: hadoop-2.7.0

// Step 3
cd /opt/mapr/hadoop/
tar -czvf hadoop-2.7.0.framework2.tar.gz hadoop-2.7.0

// Step 4
cp hadoop-2.7.0.framework2.tar.gz maprfs:///dist/hadoop-2.7.0.framework2.tar.gz