You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2017/12/08 16:46:01 UTC
[jira] [Commented] (AMBARI-22613) Hive Queries Failing with Missing
SnappyCodec Compression Type Due to Hive's Use of MapReduce Admin Env
Property
[ https://issues.apache.org/jira/browse/AMBARI-22613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283815#comment-16283815 ]
Hudson commented on AMBARI-22613:
---------------------------------
SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #8509 (See [https://builds.apache.org/job/Ambari-trunk-Commit/8509/])
AMBARI-22613 - Hive Queries Failing with Missing SnappyCodec Compression (jhurley: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=348bdd668ecf6964e2c021aeddf547b3f9bb23bf])
* (edit) ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py
* (edit) ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/configuration-mapred/mapred-site.xml
* (edit) ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml
* (edit) ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
* (edit) ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml
> Hive Queries Failing with Missing SnappyCodec Compression Type Due to Hive's Use of MapReduce Admin Env Property
> ----------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-22613
> URL: https://issues.apache.org/jira/browse/AMBARI-22613
> Project: Ambari
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Dileep Kumar Chiguruvada
> Assignee: Jonathan Hurley
> Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22613.patch
>
>
> Many queries, similar to the below {{InsertOverwrite}}, are failing after performing an HDP->HDP upgrade. The cause is that Hive does not localize the Tez TAR file, but instead uses the MapReduce2 property {{mapreduce.admin.user.env}}
> {code}
> >>> analyze table studenttab10k compute statistics for columns age ;
> INFO : Session is already open
> INFO : Dag name: analyze table studenttab10k compute st...age(Stage-0)
> INFO : Status: Running (Executing on YARN cluster with App id application_1512568339263_0003)
> [2K--------------------------------------------------------------------------------
> [2K[36;1m VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
> [22;0m[2K--------------------------------------------------------------------------------
> [2KMap 1 RUNNING 1 0 1 0 2 0
> Reducer 2 INITED 1 0 0 1 0 0
> [2K--------------------------------------------------------------------------------
> [2K[31;1mVERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 9.38 s
> [22;0m[2K--------------------------------------------------------------------------------
> [8A[2K--------------------------------------------------------------------------------
> [2K[36;1m VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
> [22;0m[2K--------------------------------------------------------------------------------
> [2KMap 1 RUNNING 1 0 0 1 4 0
> Reducer 2 INITED 1 0 0 1 0 0
> [2K--------------------------------------------------------------------------------
> [2K[31;1mVERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 14.37 s
> [22;0m[2K--------------------------------------------------------------------------------
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1512568339263_0003_2_00, diagnostics=[Task failed, taskId=task_1512568339263_0003_2_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
> at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:117)
> at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:143)
> at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:138)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149)
> ... 14 more
> Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
> at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
> at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
> at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:208)
> ... 18 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)