You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Pavas Garg (JIRA)" <ji...@apache.org> on 2017/07/19 15:29:00 UTC
[jira] [Created] (IMPALA-5680) Avro scanner still crashing when
HDFS seek fails
Pavas Garg created IMPALA-5680:
----------------------------------
Summary: Avro scanner still crashing when HDFS seek fails
Key: IMPALA-5680
URL: https://issues.apache.org/jira/browse/IMPALA-5680
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.8.0
Reporter: Pavas Garg
We had a previous Impala Jira https://issues.apache.org/jira/browse/IMPALA-4223
Impala daemon didn't crash but stopped working.
I0712 16:21:25.917135 11642 status.cc:114] Error seeking to 536870912 in file: hdfs://nameservice1/envs/production/XXX/data/XXX_ETL-f2h/xxx_logs/dt=2017-07-12/000000_0
Error(255): Unknown error 255
@ 0x8377f9 (unknown)
@ 0xa606fb (unknown)
@ 0xa5bbeb (unknown)
@ 0xa5c064 (unknown)
@ 0xbd2cd9 (unknown)
@ 0xbd36b4 (unknown)
@ 0xe1b65a (unknown)
@ 0x3a3b007851 (unknown)
@ 0x3a3a8e894d (unknown)
I0712 16:21:25.917220 30331 runtime-state.cc:217] Error from query 37402db5e53ce683:96f60bf400000000: Problem parsing file hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000000_0 at 536859282
I0712 16:21:25.917259 30331 runtime-state.cc:217] Error from query 37402db5e53ce683:96f60bf400000000: Error seeking to 536870912 in file: hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000000_0
Error(255): Unknown error 255
I0712 16:21:25.928112 11640 status.cc:114] Error seeking to 536870912 in file: hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0
Error(255): Unknown error 255
@ 0x8377f9 (unknown)
@ 0xa606fb (unknown)
@ 0xa5bbeb (unknown)
@ 0xa5c064 (unknown)
@ 0xbd2cd9 (unknown)
@ 0xbd36b4 (unknown)
@ 0xe1b65a (unknown)
@ 0x3a3b007851 (unknown)
@ 0x3a3a8e894d (unknown)
I0712 16:21:25.928170 30332 runtime-state.cc:217] Error from query 37402db5e53ce683:96f60bf400000000: Problem parsing file hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0 at 536867240
I0712 16:21:25.928185 30332 runtime-state.cc:217] Error from query 37402db5e53ce683:96f60bf400000000: Error seeking to 536870912 in file: hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0
Error(255): Unknown error 255
I0712 17:09:47.028455 11343 authentication.cc:499] Registering impala/HOSTNAME@DOMAIN.COM, keytab file /var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala.keytab
---
From HDFS namenode logs:
---
2017-07-12 16:20:43,552 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-10000/_tmp.000002_0. BP-707861034-10.13.68.11-1367011211601 blk_1319449009_1099897417390{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e5b8a4fb-a840-4de2-bd5c-d0de019cb2c8:NORMAL:10.13.68.32:1004|XXX], ReplicaUnderConstruction[[DISK]DS-5373c471-3289-4563-bff1-65b01c7c906b:NORMAL:10.13.68.30:1004|XXX], ReplicaUnderConstruction[[DISK]DS-e5988516-2067-4339-b465-f5180869f73b:NORMAL:10.13.68.28:1004|XXX]]}
2017-07-12 16:21:21,599 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-10000/_tmp.000002_0 is closed by DFSClient_attempt_1499250888632_48550_r_000002_0_-1168871812_1
2017-07-12 16:21:21,729 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/-ext-10001/tmpstats-2 is closed by DFSClient_attempt_1499250888632_48550_r_000002_0_261413855_1
2017-07-12 16:21:24,269 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.000002_0. BP-707861034-10.13.68.11-1367011211601 blk_1319449246_1099897417627{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-15ccd60f-6356-43b0-ae31-11989433f66f:NORMAL:10.13.68.46:1004|XXX], ReplicaUnderConstruction[[DISK]DS-d40c69bb-7882-42de-a0a7-4a784f4aa14a:NORMAL:10.13.68.73:1004|XXX], ReplicaUnderConstruction[[DISK]DS-395d3bce-2c3e-4922-9d94-b182dacbd14a:NORMAL:10.13.68.33:1004|XXX]]}
2017-07-12 16:21:24,396 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.000002_0 is closed by DFSClient_attempt_1499250888632_48578_m_000002_0_-1637609232_1
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)