Upgrading Spark on YARN

About this task

The following instructions explain how to upgrade an existing installation of Spark 1.x. Spark will be installed in a new subdirectory under /opt/mapr/spark.

Procedure

  1. Update repositories. MapR's rpm and deb repositories always contain the Spark version recommended for the MapR core release associated with that repository. You can connect to an internet repository or prepare a local repository with any version of Spark you need. You can also manually download packages. If you plan to install from a repository, complete the following steps each node where Spark is installed:
    1. Verify that the repository is configured correctly. See Preparing Packages and Repositories for information about setting up your ecosystem repository.
    2. Update the repository cache.
      On RedHat and CentOS
      yum clean all
      On Ubuntu
      apt-get update
  2. Back up any custom configuration files in your Spark environment. These cannot be upgraded automatically. For example, if Spark SQL is configured to work with Hive, copy the /opt/mapr/spark/spark-<version>/conf/hive-site.xml file to a backup directory.
  3. Shut down the spark-historyserver services (if the spark-historyserver is running): 
    maprcli node services -nodes <node-ip> -name spark-historyserver -action stop
  4. Install the Spark packages.
    NOTE: You only need to upgrade the mapr-spark-historyserver if your previous installation included this package.
    On Ubuntu
    apt-get install mapr-spark mapr-spark-historyserver
    On RedHat / CentOS
    yum update mapr-spark mapr-spark-historyserver
  5. Run configure.sh: 
    /opt/mapr/server/configure.sh -R
  6. Migrate Custom Configurations (optional).
    Migrate any custom configuration settings into the new default files in the conf directory(/opt/mapr/spark/spark-<version>/conf).
  7. Start spark-historyserver services (if installed): 
    
maprcli node services -nodes <node-ip> -name spark-historyserver -action start