Streaming Injest: Unable to acquire transaction batch on end point

["Grant Overby (groverby)" <gr...@cisco.com>]

I’m trying to get hive streaming ingest working with hive 1.1.0 .

The streaming ingest docs (https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest) state:

 “Partition creation being an atomic action, multiple clients can race to create the partition, but only one will succeed, so streaming clients do not have to synchronize when creating a partition”

However, my client seems to be receiving exceptions that are cause by a race on partition creation. Is this a bug? Best way to handle it?



I’m getting the following error from the client:


java.lang.RuntimeException: org.apache.hive.hcatalog.streaming.TransactionBatchUnAvailable: Unable to acquire transaction batch on end point: {metaStoreUri='thrift://twig06.twigs:9083', database='events', table='connection_events', partitionVals=[1431961200] }

        at com.cisco.com.google.common.base.Throwables.propagate(Throwables.java:160) ~[stormjar.jar:na]

        at com.cisco.tinderbox.burner.trident.functions.HiveSink.onPrepare(HiveSink.java:98) ~[stormjar.jar:na]

        at com.cisco.tinderbox.burner.trident.functions.FlushingFunction.prepare(FlushingFunction.java:74) ~[stormjar.jar:na]

        at storm.trident.planner.processor.EachProcessor.prepare(EachProcessor.java:54) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]

        at storm.trident.planner.SubtopologyBolt.prepare(SubtopologyBolt.java:121) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]

        at storm.trident.topology.TridentBoltExecutor.prepare(TridentBoltExecutor.java:231) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]

        at backtype.storm.daemon.executor$fn__4641$fn__4654.invoke(executor.clj:732) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]

        at backtype.storm.util$async_loop$fn__551.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.4.2-2.jar:0.9.3.2.2.4.2-2]

        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]

        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

Caused by: org.apache.hive.hcatalog.streaming.TransactionBatchUnAvailable: Unable to acquire transaction batch on end point: {metaStoreUri='thrift://twig06.twigs:9083', database='events', table='connection_events', partitionVals=[1431961200] }

        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:508) ~[stormjar.jar:na]

        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:458) ~[stormjar.jar:na]

        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatchImpl(HiveEndPoint.java:345) ~[stormjar.jar:na]

        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatch(HiveEndPoint.java:325) ~[stormjar.jar:na]

        at com.cisco.tinderbox.burner.trident.functions.HiveSink.createConnection(HiveSink.java:161) ~[stormjar.jar:na]

        at com.cisco.tinderbox.burner.trident.functions.HiveSink.onPrepare(HiveSink.java:96) ~[stormjar.jar:na]

        ... 8 common frames omitted

Caused by: org.apache.thrift.TApplicationException: Internal error processing open_txns

        at org.apache.thrift.TApplicationException.read(TApplicationException.java:108) ~[stormjar.jar:na]

        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71) ~[stormjar.jar:na]

        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_open_txns(ThriftHiveMetastore.java:3709) ~[stormjar.jar:na]

        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.open_txns(ThriftHiveMetastore.java:3696) ~[stormjar.jar:na]

        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxns(HiveMetaStoreClient.java:1714) ~[stormjar.jar:na]

        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.openTxnImpl(HiveEndPoint.java:517) ~[stormjar.jar:na]

        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:501) ~[stormjar.jar:na]

        ... 13 common frames omitted

The hive metastore log shows the following:


Insert of object "org.apache.hadoop.hive.metastore.model.MPartition@47ca2f1a" using statement "INSERT INTO `PARTITIONS` (`PART_ID`,`TBL_ID`,`CREATE_TIME`,`LAST_ACCESS_TIME`,`SD_ID`,`PART_NAME`) VALUES (?,?,?,?,?,?)" failed : Duplicate entry 'dt=1431961200-1' for key 'UNIQUEPARTITION'

org.datanucleus.exceptions.NucleusDataStoreException: Insert of object "org.apache.hadoop.hive.metastore.model.MPartition@47ca2f1a" using statement "INSERT INTO `PARTITIONS` (`PART_ID`,`TBL_ID`,`CREATE_TIME`,`LAST_ACCESS_TIME`,`SD_ID`,`PART_NAME`) VALUES (?,?,?,?,?,?)" failed : Duplicate entry 'dt=1431961200-1' for key 'UNIQUEPARTITION'

dt is the single partitioned column. It’s value is 1431961200. I’m unsure what appended the ‘-1’.


[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]

Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910






[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.

This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.

Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.