Re: Hive Hook - Create table xxx AS - Hive Hook BUG

[Russell Anderson <rg...@us.ibm.com>]

The following SQL causes the Hive Hook to fail with the error below:

create table bigsql.missy999 as select * from bigsql.brancha;

Question: Is this a new SEVERE BUG in Atlas7rc2 Or has this bug been fixed
in .8 but not in .7rc2 ?


Excerpt from /var/log/hive/hiveserver2.log
==========================================

2016-12-29 11:52:10,787 INFO  bridge.HiveMetaStoreBridge
(HiveMetaStoreBridge.ja\
va:createOrUpdateDBInstance(162)) - Importing objects from databaseName :
bigsql
2016-12-29 11:52:10,787 INFO  metastore.HiveMetaStore
(HiveMetaStore.java:logInf\
o(746)) - 6: get_table : db=bigsql tbl=missy999
2016-12-29 11:52:10,787 INFO  HiveMetaStore.audit
(HiveMetaStore.java:logAuditEv\
ent(371)) - ugi=admin   ip=unknown-ip-addr      cmd=get_table : db=bigsql
tbl=mi\
ssy999
2016-12-29 11:52:10,918 ERROR metadata.Hive (Hive.java:getTable(1119)) -
Table m\
issy999 not found: bigsql.missy999 table not found
2016-12-29 11:52:10,919 ERROR hook.HiveHook (HiveHook.java:run(184)) -
Atlas hoo\
k failed due to error
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
missy9\
99
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1120)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1090)
        at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities
(HiveHook.j\
ava:503)
        at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities
(HiveHook.j\
ava:524)
        at org.apache.atlas.hive.hook.HiveHook.processHiveEntity
(HiveHook.java:6\
05)
        at org.apache.atlas.hive.hook.HiveHook.registerProcess
(HiveHook.java:585\
)
        at org.apache.atlas.hive.hook.HiveHook.fireAndForget
(HiveHook.java:223)
        at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:78)
        at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182)
        at java.util.concurrent.Executors$RunnableAdapter.call
(Executors.java:51\
1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.\
java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor\
.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-12-29 11:52:11,221 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd
(148)) -\
 </PERFLOG method=serializePlan start=1483041130037 end=1483041131221
duration=1\
184 from=org.apache.hadoop.hive.ql.exec.Utilities>
2016-12-29 11:52:11,348 ERROR mr.ExecDriver (ExecDriver.java:execute(400))
- yar\
n
2016-12-29 11:52:13,528 INFO  impl.TimelineClientImpl
(TimelineClientImpl.java:s\
erviceInit(296)) - Timeline service address:
http://biginsights.ibm.com:8188/ws/\
v1/timeline/
2016-12-29 11:52:13,641 INFO  client.RMProxy (RMProxy.java:createRMProxy
(98)) - \
Connecting to ResourceManager at biginsights.ibm.com/192.168.1.132:8050
2016-12-29 11:52:14,269 INFO  fs.FSStatsPublisher
(FSStatsPublisher.java:init(49\
)) - created :
hdfs://biginsights.ibm.com:8020/apps/hive/warehouse/bigsql.db/.hi\
ve-staging_hive_2016-12-29_11-52-08_283_558364161490028907-1/-ext-10002


Russell G. Anderson
 Senior Technical Consultant




From:	Hemanth Yamijala <yh...@gmail.com>
To:	dev@atlas.incubator.apache.org
Cc:	Russell Anderson/Worcester/IBM@IBMUS, Barry
            Rosen/Worcester/IBM@IBMUS
Date:	12/29/2016 11:17 AM
Subject:	Re: Hive Hook - what is missing ? [ RESOLVED }



No worries. Glad it helped.

On 27-Dec-2016 21:18, "Russell Anderson" <rg...@us.ibm.com> wrote:

Hi Memanth Yamijala,

The magic of your thoughts !!!

So I did as you suggested but then I noticed a couple of things that hadn't
seemed important at the time :

1) In my Hadoop there were multiple entries for PRE, and FAILURE for Hive
Hook. These were left as default - so i changed these to match the one
recommended in the documentation.

2) I re-copied the libraries i had built where my AUX_PATH was pointing to.

3) I changed the permissions to match all the other jars in the hive/lib
directory - which were 'root'.

4) I then carefully examined the var/log/hive/hiveserver2.log for Hive Hook
messages

I now have the Hive Hook working as desired !!!

Thank you for getting me to look and think about things....!!!

Russ.



[image: Inactive hide details for Hemanth Yamijala ---12/26/2016 08:11:29
PM---Hi, A few questions to help debug:]Hemanth Yamijala ---12/26/2016
08:11:29 PM---Hi, A few questions to help debug:

From: Hemanth Yamijala <yh...@gmail.com>
To: dev@atlas.incubator.apache.org
Date: 12/26/2016 08:11 PM
Subject: Re: Hive Hook - what is missing ?
------------------------------



Hi,

A few questions to help debug:

1) Are you using Hive CLI or Beeline? Best results would be to
configure these hooks with HiveServer2 and use Beeline.

2) Could you please check hive logs and look for AtlasHook related
messages?

Thanks
Hemanth

On Mon, Dec 26, 2016 at 8:59 PM, Russell Anderson <rg...@us.ibm.com> wrote:
>
>
> Hi dev list,
>
> I have built the Atlas7rc2 - have working the Hive Import, the Dashboard,
> and generally things appear to be working as expected.
>
> However, I have followed the instruction below but cannot seem to get the
> Hive Hook to detect table hive creations to be added to the metadata /
> lineage.
>
> Can anyone suggest how to debug what could be missing regardless of
> following the instructions below ? Any and all help appreciated !!!
>
> Regards,
>
> Russ.
>
> ============================================================
=================
>
> Hive Hook
>
>
> Hive supports listeners on hive command execution using hive hooks. This
is
> used to add/update/remove entities in Atlas using the model defined in
> org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the
> request to a thread pool executor to avoid blocking the command
execution.
> The thread submits the entities as message to the notification server and
> atlas server reads these messages and registers the entities. Follow
these
> instructions in your hive set-up to add hive hook for Atlas:
>       Set-up atlas hook in hive-site.xml of your hive configuration:
>     <property>
>       <name>hive.exec.post.hooks</name>
>       <value>org.apache.atlas.hive.hook.HiveHook</value>
>     </property>
>
>
>     <property>
>       <name>atlas.cluster.name</name>
>       <value>primary</value>
>     </property>
>
>
>       Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in
>       hive-env.sh of your hive configuration
>       Copy <atlas-conf>/atlas-application.properties to the hive conf
>       directory.
>
>
> The following properties in <atlas-conf>/atlas-application.properties
> control the thread pool and notification details:
>       atlas.hook.hive.synchronous - boolean, true to run the hook
>       synchronously. default false. Recommended to be set to false to
avoid
>       delays in hive query completion.
>       atlas.hook.hive.numRetries - number of retries for notification
>       failure. default 3
>       atlas.hook.hive.minThreads - core number of threads. default 5
>       atlas.hook.hive.maxThreads - maximum number of threads. default 5
>       atlas.hook.hive.keepAliveTime - keep alive time in msecs. default
10
>       atlas.hook.hive.queueSize - queue size for the threadpool. default
>       10000