You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2016/09/08 02:42:21 UTC
[jira] [Commented] (KUDU-1403) Investigate create table test
failure from Impala
[ https://issues.apache.org/jira/browse/KUDU-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472506#comment-15472506 ]
Todd Lipcon commented on KUDU-1403:
-----------------------------------
[~mjacobs] have you seen this in recent months? If not, I'll close as Cannot Repro
> Investigate create table test failure from Impala
> -------------------------------------------------
>
> Key: KUDU-1403
> URL: https://issues.apache.org/jira/browse/KUDU-1403
> Project: Kudu
> Issue Type: Task
> Components: tserver
> Affects Versions: 0.7.0
> Reporter: Casey Ching
> Attachments: logs.zip
>
>
> Impala has a test that creates, reads, and deletes a Kudu table. The test usually passes but it failed recently. Adar looked at this logs and suggested filing an issue.
> Test: https://github.com/cloudera/Impala/blob/7f23df9a72941dab5fa54a41bf58ff8f80fc584f/tests/query_test/test_kudu.py#L191
> The Kudu version was 0.7.
> I'll try to attach the logs.
> The failure message was
> {noformat}
> 15:02:51 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 1 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] PASSED
> 15:04:49 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 10 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] PASSED
> 15:06:45 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 0 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/gzip/block] PASSED
> 15:08:43 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 0 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/bzip/block] PASSED
> 15:10:39 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 10 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/gzip/block] PASSED
> 15:12:45 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 1 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/bzip/block] ERROR
> 15:13:08 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 1 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/snap/block] ERROR
> 15:13:28 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 10 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/snap/block] ERROR
> 15:13:46 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 0 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/lzo/block] ERROR
> 15:14:00 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 0 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/none] ERROR
> 15:14:14 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 10 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/lzo/block] ERROR
> 15:14:28 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 1 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/none] ERROR
> 15:14:42 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 1 | exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/def/block] ERROR
> 15:14:56 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 10 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/def/block] ERROR
> 15:15:10 query_test/test_kudu.py::TestKuduMemLimits::test_low_mem_limit_low_selectivity_scan[mem_limit: 0 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: seq/def/record] ERRORTraceback (most recent call last):
> 15:15:24 File "/usr/lib64/python2.7/logging/__init__.py", line 872, in emit
> 15:15:24 stream.write(fs % msg)
> 15:15:24 File "/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/infra/python/env/lib/python2.7/site-packages/_pytest/capture.py", line 229, in write
> 15:15:24 self.buffer.write(obj)
> 15:15:24 ValueError: I/O operation on closed file
> 15:15:24 Logged from file impala_connection.py, line 152
> 15:15:24
> 15:15:24
> 15:15:24 ==================================== ERRORS ====================================
> 15:15:24 ERROR at setup of TestKuduMemLimits.test_low_mem_limit_low_selectivity_scan[mem_limit: 1 | exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/bzip/block]
> 15:15:24 query_test/test_kudu.py:160: in setup_method
> 15:15:24 self.client.execute(self.CREATE)
> 15:15:24 common/impala_connection.py:161: in execute
> 15:15:24 return self.__beeswax_client.execute(sql_stmt, user=user)
> 15:15:24 beeswax/impala_beeswax.py:163: in execute
> 15:15:24 handle = self.__execute_query(query_string.strip(), user=user)
> 15:15:24 beeswax/impala_beeswax.py:327: in __execute_query
> 15:15:24 handle = self.execute_query_async(query_string, user=user)
> 15:15:24 beeswax/impala_beeswax.py:323: in execute_query_async
> 15:15:24 return self.__do_rpc(lambda: self.imp_service.query(query,))
> 15:15:24 beeswax/impala_beeswax.py:448: in __do_rpc
> 15:15:24 raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> 15:15:24 E ImpalaBeeswaxException: ImpalaBeeswaxException:
> 15:15:24 E INNER EXCEPTION: <class 'impala._thrift_gen.beeswax.ttypes.BeeswaxException'>
> 15:15:24 E MESSAGE:
> 15:15:24 E ImpalaRuntimeException: Error creating Kudu table
> 15:15:24 E CAUSED BY: NonRecoverableException: Too many attempts: KuduRpc(method=IsCreateTableDone, tablet=null, attempt=6, DeadlineTracker(timeout=10000, elapsed=7650), Deferred@1729211867(state=PENDING, result=null, callback=(continuation of Deferred@967653818 after org.kududb.client.AsyncKuduClient$4@1ba8ffeb@464060395) -> (continuation of Deferred@1949936508 after org.kududb.client.AsyncKuduClient$5@76299580@1982436736) -> (continuation of Deferred@605438747 after org.kududb.client.AsyncKuduClient$5@318fb41f@831501343) -> (continuation of Deferred@1366980772 after org.kududb.client.AsyncKuduClient$5@3f8680c6@1065779398) -> (continuation of Deferred@393242559 after org.kududb.client.AsyncKuduClient$5@54ab389e@1420507294), errback=(continuation of Deferred@967653818 after org.kududb.client.AsyncKuduClient$4@1ba8ffeb@464060395) -> (continuation of Deferred@1949936508 after org.kududb.client.AsyncKuduClient$5@76299580@1982436736) -> (continuation of Deferred@605438747 after org.kududb.client.AsyncKuduClient$5@318fb41f@831501343) -> (continuation of Deferred@1366980772 after org.kududb.client.AsyncKuduClient$5@3f8680c6@1065779398) -> (continuation of Deferred@393242559 after org.kududb.client.AsyncKuduClient$5@54ab389e@1420507294)))
> {noformat}
> So the test passed several times using various mem limits, then started failing repeatedly.
> Some of the interesting things Adar found in the logs:
> 1) File system syncs are slow, ~5s in some cases.
> 2) Remote boot strapping was attempted repeatedly for the tablet and requested from the same peer. The repeat timing was something like 0.5 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)