Spark DataFrame.registerTempTable doesn't update the temp table in jdbc hive
I am new to spark. we are trying to use Hive JDBC to access temp table that registered by RDD from Spark. We can get the RDD working for the first time by doing code as follows:
dt.registerTempTable("Table1"); dt.cache(); HiveContext hc = HiveContextManager.getInstance(); // a singlton HiveContext instance hc.registerDataFrameAsTable(dt, "Table1"); hc.cacheTable("Table1"); dt.cache(); dt.collect(); HiveThriftServer2.startWithContext(hc);
However, when we get new RDD data coming in and try to update the temp table, I cannot see the new record in Hive JDBC (running "select count(*) from Table1" in the jdbc:hive2 command line gives the same number). The code is as follows:
DataFrame tempDT = currentDT.unionAll(newDT); HiveContext hc = HiveContextManager.getInstance(); hc.uncacheTable("Table1"); tempDT.registerTempTable("Table1"); hc.cacheTable("Table1");
Anyone has any idea? Please help. Thanks a lot.
It turns out RegisterTempTable works perfect. It is some issue during the time when we create the newDT. Thanks.