Spark 3.3.2.100 - 2307 (EEP 9.1.2) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 3.3.2.100.

The notes below relate specifically to the Hewlett Packard Enterprise Distribution for Apache Hadoop. For more information, you may also want to consult the open-source Spark 3.3.2 Release Notes.

These release notes contain only Hewlett Packard Enterprise specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 3.3.2.100
Release Date July 2023
HPE Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 3.3.2.100-eep-2307
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.

Hive Support

  • Starting from Spark 3.1.2, Spark supports Hive 2.3.

Delta Lake Support

Spark 3.2.0 and later provides Delta Lake support on HPE Ezmeral Data Fabric. See Apache Spark Feature Support.

New in This Release

Fixes

This HPE release includes the following new fixes since the latest Spark release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
41a0e26 05/06/2023 MapR [SPARK-1185] Update Spark dep on EEP 9.1.2 components artifacts
1988b41 06/06/2023 MapR [SPARK-1188] [Java 17] Need to open java.lang module for Spark HS, Spark Master and Spark Workers
28948bd 19/06/2023 MapR [SPARK-1190] Address CVE-2022-37865
fef9cec 21/06/2023 MapR [SPARK-1195] Fix Spark scripts as htrace-core*.jar was removed from HBase
7a5cfdf 08/07/2023 [MAPRYARN-397] Proxy should respond to client with a redirect if it gets SSL errors from server
cf50fbd 11/07/2023 MapR [SPARK-1196] MaprDB spark connector - Cannot convert true to a MapRDB predicate
ca1ae41 12/07/2023 MapR [SPARK-1202] Configure.sh overwrites the hive-site.xml

Known Issues and Limitations

  • When you enable the SSL in a mixed (FIPS and non-FIPS) configuration, Spark application run fails. To run Spark applications, set spark.ssl.ui.enabled option to false in spark-defaults.conf configuration file.

  • SPARK-1099: Non-mapr user is unable to insert values into Hive table by using Spark Thrift Server
    Symptoms:
    Navigate to Spark Beeline as a non-mapr user and connect to Spark Thrift Server.
    !connect jdbc:hive2://<node1.cluster.com>:2304/default;ssl=true;auth=maprsasl
    Create a table:
    CREATE TABLE nonmaprctastest2 (key int);
insert into table nonmaprctastest2 values 1, 2, 3;
    The following error occurs:
    Caused by: java.lang.RuntimeException: Cannot create staging directory: 'maprfs:/user/hive/warehouse/nonmaprctastest2/.hive-staging_hive_2022-08-23_11-38-31_177_3217175113512758641-4': User mapruser1(user id 5001) has been denied access to create .hive-staging_hive_2022-08-23_11-38-31_177_3217175113512758641-4
    Cause:
    In Hive 2.x, permissions for all the tables in maprfs:///user/hive/warehouse/ directory are set to 777. However, in Hive 3.x, permissions for table directories are set to 755. In EEP, Spark Thrift Server creates the table as a user who started the Spark Thrift Server. When Hive 3.x changes the user to the user who did not start he Spark Thrift Server, the user can no longer make write operation with tables.
    Workaround:
    You can choose one of the following workarounds:
    • After creating the Hive table, set permissions to 777 in maprfs:///user/hive/warehouse directory.
    • After creating the Hive table, set owner to the user who created the Hive table.
    • Use HiveServer2 instead of Spark Thrift Server which uses impersonation.

Resolved Issues

  • None.