[jira] [Updated] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.

["Mithun Radhakrishnan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mithun Radhakrishnan updated HIVE-6409:
---------------------------------------

    Attachment: HIVE-6409.trunk.patch

Removed the finally{} block from commitJob(). If commit succeeds, there are no exceptions, and the tokens are cancelled. If commit fails, the cancel is avoided, and abortJob() takes care of cancelling the token.

I've also removed an annoying typo. I would also like to remove the following  catch blocks:
{code}
catch(TException e){} 
{code}

TException isA MetaException, so this is redundant. The error handling is identical in both cases. I'd like to remove this cruft, if it's alright.

> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-6409
>                 URL: https://issues.apache.org/jira/browse/HIVE-6409
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>    Affects Versions: 0.12.0, 0.13.0
>            Reporter: Mithun Radhakrishnan
>            Assignee: Mithun Radhakrishnan
>         Attachments: HIVE-6409.trunk.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls the underlying OutputCommitter and then attempts to register partitions in HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels delegation tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
>     at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
>     at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
>     at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:601)
>     at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
>     ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are unbalanced
> calls to openTransaction/commitTransaction)
>     at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run, thus yielding the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password
> 	at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
> 	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> 	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> 	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> 	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> 	at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob() has succeeded.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)