You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2022/09/18 02:24:00 UTC
[jira] [Created] (IMPALA-11593) TestMemReservations.test_per_backend_min_reservation failed by NullPointerException from libhdfs in S3 builds
Quanlong Huang created IMPALA-11593:
---------------------------------------
Summary: TestMemReservations.test_per_backend_min_reservation failed by NullPointerException from libhdfs in S3 builds
Key: IMPALA-11593
URL: https://issues.apache.org/jira/browse/IMPALA-11593
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
Saw this failure on an S3 build:
{noformat}
custom_cluster/test_mem_reservations.py:102: in test_per_backend_min_reservation
assert t.error is None
E assert 'ImpalaBeeswaxException:\n Query aborted:Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera....warehouse/alltypes/year=2009/month=9/090901.txt\nError(255): Unknown error 255\nRoot cause: NullPointerException: \n\n' is None
E + where 'ImpalaBeeswaxException:\n Query aborted:Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera....warehouse/alltypes/year=2009/month=9/090901.txt\nError(255): Unknown error 255\nRoot cause: NullPointerException: \n\n' = <QuerySubmitThread(Thread-165, stopped 140272709113600)>.error
{noformat}
Impalad logs for the query:
{noformat}
I0915 03:12:33.839942 21677 impala-server.cc:1333] 09439d05a2468038:3816f0f200000000] Registered query query_id=09439d05a2468038:3816f0f200000000 session_id=874c5100c59607af:a86e04c8f62bb9a9
I0915 03:12:33.889168 21677 Frontend.java:1628] 09439d05a2468038:3816f0f200000000] Analyzing query: select max(t.c1), avg(t.c2), min(t.c3), avg(c4), avg(c5), avg(c6)
from (select
max(tinyint_col) over (order by int_col) c1,
avg(tinyint_col) over (order by smallint_col) c2,
min(tinyint_col) over (order by smallint_col desc) c3,
rank() over (order by int_col desc) c4,
dense_rank() over (order by bigint_col) c5,
first_value(tinyint_col) over (order by bigint_col desc) c6
from functional.alltypes) t; db: default
I0915 03:12:33.981251 21677 FeSupport.java:315] 09439d05a2468038:3816f0f200000000] Requesting prioritized load of table(s): functional.alltypes
I0915 03:12:33.986737 21677 thrift-util.cc:99] 09439d05a2468038:3816f0f200000000] TSocket::open() connect() <Host: localhost Port: 26000>: Connection refused
I0915 03:12:34.582643 21677 BaseAuthorizationChecker.java:113] 09439d05a2468038:3816f0f200000000] Authorization check took 693 ms
I0915 03:12:34.582674 21677 Frontend.java:1671] 09439d05a2468038:3816f0f200000000] Analysis and authorization finished.
I0915 03:12:34.723712 21208 control-service.cc:148] 4a4ebd3b7575254c:eb71cd8000000000] ExecQueryFInstances(): query_id=4a4ebd3b7575254c:eb71cd8000000000 coord=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27000 #instances=1
I0915 03:12:34.738032 21758 query-state.cc:942] 4a4ebd3b7575254c:eb71cd8000000002] Executing instance. instance_id=4a4ebd3b7575254c:eb71cd8000000002 fragment_idx=1 per_fragment_instance_idx=1 coord_state_idx=1 #in-flight=1
I0915 03:12:34.850791 21820 admission-controller.cc:1819] 09439d05a2468038:3816f0f200000000] Trying to admit id=09439d05a2468038:3816f0f200000000 in pool_name=default-pool executor_group_name=default per_host_mem_estimate=1.34 GB dedicated_coord_mem_estimate=1.10 GB max_requests=-1 max_queued=200 max_mem=-1.00 B
I0915 03:12:34.850811 21820 admission-controller.cc:1827] 09439d05a2468038:3816f0f200000000] Stats: agg_num_running=1, agg_num_queued=0, agg_mem_reserved=1.56 GB, local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=192.46 MB, topN_query_stats: queries=[4a4ebd3b7575254c:eb71cd8000000000], total_mem_consumed=192.46 MB, fraction_of_pool_total_mem=1; pool_level_stats: num_running=1, min=192.46 MB, max=192.46 MB, pool_total_mem=192.46 MB, average_per_query=192.46 MB)
I0915 03:12:34.850852 21820 admission-controller.cc:1218] 09439d05a2468038:3816f0f200000000] Admitting query id=09439d05a2468038:3816f0f200000000
I0915 03:12:34.850939 21820 impala-server.cc:2159] 09439d05a2468038:3816f0f200000000] Registering query locations
I0915 03:12:34.850998 21820 coordinator.cc:150] 09439d05a2468038:3816f0f200000000] Exec() query_id=09439d05a2468038:3816f0f200000000 stmt=select max(t.c1), avg(t.c2), min(t.c3), avg(c4), avg(c5), avg(c6)
from (select
max(tinyint_col) over (order by int_col) c1,
avg(tinyint_col) over (order by smallint_col) c2,
min(tinyint_col) over (order by smallint_col desc) c3,
rank() over (order by int_col desc) c4,
dense_rank() over (order by bigint_col) c5,
first_value(tinyint_col) over (order by bigint_col desc) c6
from functional.alltypes) t;
I0915 03:12:34.851434 21820 coordinator.cc:474] 09439d05a2468038:3816f0f200000000] starting execution on 3 backends for query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:34.856995 21208 control-service.cc:148] 09439d05a2468038:3816f0f200000000] ExecQueryFInstances(): query_id=09439d05a2468038:3816f0f200000000 coord=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 #instances=2
I0915 03:12:34.858456 21820 coordinator.cc:533] 09439d05a2468038:3816f0f200000000] started execution on 3 backends for query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:34.860503 21841 query-state.cc:942] 09439d05a2468038:3816f0f200000002] Executing instance. instance_id=09439d05a2468038:3816f0f200000002 fragment_idx=1 per_fragment_instance_idx=1 coord_state_idx=0 #in-flight=2
I0915 03:12:34.860591 21843 query-state.cc:942] 09439d05a2468038:3816f0f200000000] Executing instance. instance_id=09439d05a2468038:3816f0f200000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=3
I0915 03:12:35.057634 21208 coordinator.cc:1032] Backend completed: host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27002 remaining=3 query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:35.057649 21208 coordinator-backend-state.cc:371] query_id=09439d05a2468038:3816f0f200000000: first in-progress backend: impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
I0915 03:12:35.149704 21208 coordinator.cc:1032] Backend completed: host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27000 remaining=2 query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:35.149719 21208 coordinator-backend-state.cc:371] query_id=09439d05a2468038:3816f0f200000000: first in-progress backend: impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
I0915 03:12:35.106189 21377 status.cc:71] Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to open HDFS file s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
@ 0x1f096f4 impala::Status::Status()
@ 0x1f1dfdb impala::Status::Status()
@ 0x2ea28bb impala::io::OpenHdfsFileOp::Execute()
@ 0x2ea3884 impala::SynchronousWorkItem::WorkerExecute()
@ 0x2ea4071 impala::SynchronousThreadPool::Worker()
@ 0x2ea6a89 boost::detail::function::void_function_invoker2<>::invoke()
@ 0x2ea6734 boost::function2<>::operator()()
@ 0x2ea56f1 impala::ThreadPool<>::WorkerThread()
@ 0x2ea8819 boost::_mfi::mf1<>::operator()()
@ 0x2ea8645 boost::_bi::list2<>::operator()<>()
@ 0x2ea82d2 boost::_bi::bind_t<>::operator()()
@ 0x2ea7eaf boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x221e4f7 boost::function0<>::operator()()
@ 0x29a898f impala::Thread::SuperviseThread()
@ 0x29b12f0 boost::_bi::list5<>::operator()<>()
@ 0x29b1214 boost::_bi::bind_t<>::operator()()
@ 0x29b11d5 boost::detail::thread_data<>::run()
@ 0x42018b1 thread_proxy
@ 0x7f4590612ea4 start_thread
@ 0x7f458d00cb0c __clone
I0915 03:12:35.195694 21854 hdfs-scan-node.cc:515] 09439d05a2468038:3816f0f200000002] Scan node (id=0) ran into a parse error for scan range s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt(0:20179). Processed 0 bytes.
I0915 03:12:35.196115 21841 query-state.cc:951] 09439d05a2468038:3816f0f200000002] Instance completed. instance_id=09439d05a2468038:3816f0f200000002 #in-flight=1 status=DISK_IO_ERROR: Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to open HDFS file s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
I0915 03:12:35.196135 21829 query-state.cc:462] 09439d05a2468038:3816f0f200000000] UpdateBackendExecState(): last report for 09439d05a2468038:3816f0f200000000
I0915 03:12:35.198632 21208 coordinator.cc:1032] Backend completed: host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 remaining=1 query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:35.198649 21208 coordinator.cc:752] ExecState: query id=09439d05a2468038:3816f0f200000000 finstance=09439d05a2468038:3816f0f200000002 on host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 (EXECUTING -> ERROR) status=Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to open HDFS file s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
I0915 03:12:35.198689 21208 coordinator-backend-state.cc:974] query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27001: Not cancelling because the backend is already done: Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to open HDFS file s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
I0915 03:12:35.198695 21208 coordinator-backend-state.cc:974] query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27000: Not cancelling because the backend is already done:
I0915 03:12:35.198702 21208 coordinator-backend-state.cc:974] query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27002: Not cancelling because the backend is already done:
I0915 03:12:35.198706 21208 coordinator.cc:999] CancelBackends() query_id=09439d05a2468038:3816f0f200000000, tried to cancel 0 backends
I0915 03:12:35.198752 21208 coordinator.cc:1375] Release admission control resources for query_id=09439d05a2468038:3816f0f200000000
{noformat}
This could be due to the same cause of IMPALA-11592. Maybe there is an issue inside the hdfs client. Hadoop version: hadoop-3.1.1.7.2.16.0-171
--
This message was sent by Atlassian Jira
(v8.20.10#820010)