You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2019/11/05 16:59:57 UTC
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14636
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
This test makes the following changes / fixes when running Impala tests
on ABFS:
* Skips some tests in test_lineage.py that don't work on ABFS / ADLS
(they were already skipped for S3)
* Skips some tests in test_mt_dop.py; the test creates a directory that
ends with a period (and ABFS does not support writing files or
directories that end with a period)
* Removes the ABFS skip flag SkipIfABFS.trash (IMPALA-7726: Drop with
purge tests fail against ABFS due to trash misbehavior"); I removed
these flags and looped the tests overnight with no failures, so it is
likely whatever bug was causing this has now been fixed
* Now that HADOOP-15860 has been resolved, and the agreed upon behavior
for ABFS is that it will fail if a client tries to write a file /
directory that ends with a period, I added a new entry to the SkipIfABFS
class called file_or_folder_name_ends_with_period and applied it where
necessary
Testing:
* Ran core tests on ABFS
Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
---
M tests/common/skip.py
M tests/custom_cluster/test_lineage.py
M tests/custom_cluster/test_mt_dop.py
M tests/metadata/test_ddl.py
M tests/query_test/test_insert.py
5 files changed, 8 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/14636/1
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2019 05:44:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5176/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2019 01:25:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4948/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Nov 2019 17:51:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
This test makes the following changes / fixes when running Impala tests
on ABFS:
* Skips some tests in test_lineage.py that don't work on ABFS / ADLS
(they were already skipped for S3)
* Skips some tests in test_mt_dop.py; the test creates a directory that
ends with a period (and ABFS does not support writing files or
directories that end with a period)
* Removes the ABFS skip flag SkipIfABFS.trash (IMPALA-7726: Drop with
purge tests fail against ABFS due to trash misbehavior"); I removed
these flags and looped the tests overnight with no failures, so it is
likely whatever bug was causing this has now been fixed
* Now that HADOOP-15860 has been resolved, and the agreed upon behavior
for ABFS is that it will fail if a client tries to write a file /
directory that ends with a period, I added a new entry to the SkipIfABFS
class called file_or_folder_name_ends_with_period and applied it where
necessary
Testing:
* Ran core tests on ABFS
Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Reviewed-on: http://gerrit.cloudera.org:8080/14636
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/common/skip.py
M tests/custom_cluster/test_lineage.py
M tests/custom_cluster/test_mt_dop.py
M tests/metadata/test_ddl.py
M tests/query_test/test_insert.py
5 files changed, 8 insertions(+), 5 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14636/1/tests/common/skip.py
File tests/common/skip.py:
http://gerrit.cloudera.org:8080/#/c/14636/1/tests/common/skip.py@78
PS1, Line 78: file_or_folder_name_ends_with_period = pytest.mark.skipif(IS_ABFS,
: reason="ABFS does not support file / directories that end with a period")
Will we still need this when we have the change in IMPALA-8557? Or is this something else?
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Nov 2019 22:26:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 1: Code-Review+2
This makes sense to me. Thanks for running the ABFS tests.
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2019 00:47:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Nov 2019 01:25:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14636/1/tests/common/skip.py
File tests/common/skip.py:
http://gerrit.cloudera.org:8080/#/c/14636/1/tests/common/skip.py@78
PS1, Line 78: file_or_folder_name_ends_with_period = pytest.mark.skipif(IS_ABFS,
: reason="ABFS does not support file / directories that end with a period")
> Yeah we will still need this even after fixing IMPALA-8557. ABFS in general
Thanks, makes sense
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Nov 2019 22:32:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14636 )
Change subject: IMPALA-9117, IMPALA-7726: Fixed a few unit tests for ABFS
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14636/1/tests/common/skip.py
File tests/common/skip.py:
http://gerrit.cloudera.org:8080/#/c/14636/1/tests/common/skip.py@78
PS1, Line 78: file_or_folder_name_ends_with_period = pytest.mark.skipif(IS_ABFS,
: reason="ABFS does not support file / directories that end with a period")
> Will we still need this when we have the change in IMPALA-8557? Or is this
Yeah we will still need this even after fixing IMPALA-8557. ABFS in general just does not allow files / folders to end in a period. There are some tests that write files / folder that end in a period: https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/insert.test#L433
--
To view, visit http://gerrit.cloudera.org:8080/14636
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18ae5b0f7de6aa7628a1efd780ff30a0cc3c5285
Gerrit-Change-Number: 14636
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Nov 2019 22:31:58 +0000
Gerrit-HasComments: Yes