Spark DataFrame.registerTempTable doesn't update the temp table in jdbc hive

I am new to spark. we are trying to use Hive JDBC to access temp table that registered by RDD from Spark. We can get the RDD working for the first time by doing code as follows:

dt.registerTempTable("Table1");
dt.cache();
HiveContext hc = HiveContextManager.getInstance(); // a singlton HiveContext instance
hc.registerDataFrameAsTable(dt, "Table1");
hc.cacheTable("Table1");
dt.cache();
dt.collect();
HiveThriftServer2.startWithContext(hc);

However, when we get new RDD data coming in and try to update the temp table, I cannot see the new record in Hive JDBC (running "select count(*) from Table1" in the jdbc:hive2 command line gives the same number). The code is as follows:

DataFrame tempDT = currentDT.unionAll(newDT);           
HiveContext hc = HiveContextManager.getInstance();      
hc.uncacheTable("Table1");
tempDT.registerTempTable("Table1");
hc.cacheTable("Table1");

Anyone has any idea? Please help. Thanks a lot.

Answers


It turns out RegisterTempTable works perfect. It is some issue during the time when we create the newDT. Thanks.


Need Your Help

Django: Relations between users (store unique pairs / combinations)

django django-models

I would like to model the relations between users. For example: do they know each-other ?

Adding more than 2 metrics to a Google Analytics

events google-analytics timeline

Is it possible to track more than two metrics in a timeline on Google Analytics?