[jira] [Commented] (HIVE-21280) Null pointer exception on running compaction against a MM table.

["Vaibhav Gumashta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786409#comment-16786409 ] 

Vaibhav Gumashta commented on HIVE-21280:
-----------------------------------------

Thanks for the patch [~aditya-shah]; I will review it tomorrow.

> Null pointer exception on running compaction against a MM table.
> ----------------------------------------------------------------
>
>                 Key: HIVE-21280
>                 URL: https://issues.apache.org/jira/browse/HIVE-21280
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 3.1.1
>            Reporter: Aditya Shah
>            Assignee: Aditya Shah
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-21280.patch
>
>
> On running compaction on MM table, got a null pointer exception while getting HDFS session path. The error seemed to me that the session state was not started for these queries. Even after making it start it further fails in running a Teztask for insert overwrite on temp table with the contents of the original table. The cause for this is Tezsession state is not able to initialize due to Illegal Argument exception being thrown at the time of setting up caller context in Tez task due to caller id which uses queryid being an empty string. 
> I do think session state needs to be started and each of the queries running for compaction (I'm also doubtful for stats updater thread's queries) should have a query id. Some details are as follows:
> Steps to reproduce:
> 1) Using beeline with HS2 and HMS
> 2) create an MM table
> 3) Insert a few values in the table
> 4) alter table mm_table compact 'major'; 
> Stack trace on HMS:
> {code:java}
> compactor.Worker: Caught exception while trying to compact id:8,dbname:default,tableName:acid_mm_orc,partName:null,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestWriteId:0. Marking failed to avoid repeated failures, java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run create temporary table default.tmp_compactor_acid_mm_orc_1550222367257(`a` int, `b` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'WITH SERDEPROPERTIES (
> 'serialization.format'='1')STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://localhost:9000/user/hive/warehouse/acid_mm_orc/_tmp_2d8a096c-2db5-4ed8-921c-b3f6d31e079e/_base' TBLPROPERTIES ('transactional'='false')
> at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runMmCompaction(CompactorMR.java:373)
> at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:241)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run create temporary table default.tmp_compactor_acid_mm_orc_1550222367257(`a` int, `b` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'WITH SERDEPROPERTIES (
> 'serialization.format'='1')STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://localhost:9000/user/hive/warehouse/acid_mm_orc/_tmp_2d8a096c-2db5-4ed8-921c-b3f6d31e079e/_base' TBLPROPERTIES ('transactional'='false')
> at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runOnDriver(CompactorMR.java:525)
> at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runMmCompaction(CompactorMR.java:365)
> ... 2 more
> Caused by: java.lang.NullPointerException: Non-local session path expected to be non-null
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:228)
> at org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:815)
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:309)
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:295)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:591)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1684)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1807)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1567)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1556)
> at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runOnDriver(CompactorMR.java:522)
> ... 3 more
> {code}
> cc: [~ekoifman] [~vgumashta] [~sershe]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)