You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Vuk Ercegovac (JIRA)" <ji...@apache.org> on 2018/10/24 18:03:00 UTC
[jira] [Commented] (IMPALA-7733)
TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
[ https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662632#comment-16662632 ]
Vuk Ercegovac commented on IMPALA-7733:
---------------------------------------
Here is another one. The direct cause differs, but fwict, the issue here is flakiness with S3. If these are truly separate issues that go beyond a flaky test, these can be separated/replaced. The first issue above is from an immediate failure when renaming. This issue below is on the subsequent query where a file is not found.
{noformat}
query_test/test_insert_parquet.py:113: in test_insert_parquet
self.run_test_case('insert_parquet', vector, unique_database, multiple_impalad=True)
common/impala_test_suite.py:467: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:688: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:170: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:182: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:356: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:377: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:Disk I/O error: Failed to open HDFS file s3a://<removed>/test-warehouse/test_insert_parquet_1332ae40.db/orders_insert_table/e44c72680633256a-6851b4d300000000_457705041_data.0.parq
E Error(2): No such file or directory
E Root cause: FileNotFoundException: No such file or directory: s3a://<removed>/test-warehouse/test_insert_parquet_1332ae40.db/orders_insert_table/e44c72680633256a-6851b4d300000000_457705041_data.0.parq{noformat}
> TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
> -------------------------------------------------------------------------
>
> Key: IMPALA-7733
> URL: https://issues.apache.org/jira/browse/IMPALA-7733
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.1.0
> Reporter: Vuk Ercegovac
> Priority: Blocker
> Labels: flaky-test
>
> I see two examples in the past two months or so where this test fails due to a rename error on S3. The test's stacktrace looks like this:
> {noformat}
> query_test/test_insert_parquet.py:112: in test_insert_parquet
> self.run_test_case('insert_parquet', vector, unique_database, multiple_impalad=True)
> common/impala_test_suite.py:408: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:625: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:176: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:350: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:371: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Error(s) moving partition files. First error (of 1) was: Hdfs op (RENAME s3a://<removed>/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq TO s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq) failed, error was: s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq
> E Error(5): Input/output error{noformat}
> Since we know this happens once in a while, some ideas to deflake it:
> * retry
> * check for this specific issue... if we think its platform flakiness, then we should skip it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org