Zeppelin fails; Class UTF8String.class is different between Vora and Spark 1.5.2 libraries
I installed Vora 1.1. Patch 1 on HDP 2.3 with Spark 1.5.2, on SLES 11 SP3. It's not precisely the configuration mentioned in the Note 2213226, but shell-version of Vora seems to be working properly with the test 2.7 of the Installation manual (the latter did't prescribe HDP versions depending on the OS version, hence I went for HDP2.3 under SLES).
I have problems with Zeppelin, though. The github installation of version 0.5.6 seems to be successful, and I can execute the "create table" statement in Zeppelin notepad, but when executing "show tables" statement I get error:
Error: Job aborted due to stage failure: Task 0 in stage 12.0 failed 4 times, most recent failure: Lost task 0.3 in stage 12.0 (TID 36, eba156.extendtec.com.au): java.io.InvalidClassException: org.apache.spark.unsafe.types.UTF8String; local class incompatible: stream classdesc serialVersionUID = 7459647620003804432, local class serialVersionUID = 7786395165093970948 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) at (blablabla)
I believe I found the reason why:
- The class UTF8String.class coming from the library spark-sap-datasources-1.2.10-assembly.jar (and then used by Zeppelin) is dated Jan 20 and has size 17919 bytes.
- The class UTF8String.class contained in the Spark's 1.5.2. library is dated Dec 16 and has size 18653
So I guess versions of these libraries do not match. How should I proceed? Thanks!
Up to Vora1.1 Patch 1 the Spark 1.5.2 version that comes with HDP2.3.4 is not officially supported (the HDP-Spark1.5.2 version is slightly different from the Apache Spark1.5.2 version). There are 2 known issues with the Thriftserver and Zeppelin. Easiest workaround is to install Apache Spark 1.5.2 outside of Ambari and not use the HDP-Spark version.
As of Vora 1.2 (released March 31 2016) both issues with the HDP-Spark 1.5.2 version are resolved and Vora is fully compatible with it.
I have copied the mentioned class over from Spark's "unbinded" library to the spark-vora-zeppelin combined one, overwriting the class there. The "SHOW TABLES" executed without any issues. I wonder whether it is the appropriate solution, but so far it worked.