You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2018/11/20 02:04:56 UTC

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners 

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11959


Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................

IMPALA-7804: Mitigate s3 consistency issues for test_scanners

test_scanners.py has seen several flaky failures on
s3 due to eventual consistency. The symptom is Impala
being unable to read a file that it just loaded to s3.

A large number of tables used in test_scanners.py
use the file_utils helper functions for creating
the tables. These follow the pattern:
1. Copy files to temporary directory in HDFS/S3/etc
2. Create table
3. Run LOAD DATA to move the files to the table

In step #3, LOAD DATA gets the metadata for the
table before it runs the move statement on the
files. Subsequent queries on the table will not
need to reload metadata and can access the file
quickly after the move.

This changes the ordering to put the files in place
before loading metadata. This may improve the
likelihood that the filesystem is consistent by
the time we read it. Specifically, we now do:
1. Put the files in directory that the table
   will use when it is created.
2. Create table
Neither of these steps load metadata, so the next
query that runs will load metadata.

Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
---
M tests/common/file_utils.py
1 file changed, 24 insertions(+), 14 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/11959/1
-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Nov 2018 01:44:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11959

to look at the new patch set (#3).

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................

IMPALA-7804: Mitigate s3 consistency issues for test_scanners

test_scanners.py has seen several flaky failures on
s3 due to eventual consistency. The symptom is Impala
being unable to read a file that it just loaded to s3.

A large number of tables used in test_scanners.py
use the file_utils helper functions for creating
the tables. These follow the pattern:
1. Copy files to temporary directory in HDFS/S3/etc
2. Create table
3. Run LOAD DATA to move the files to the table

In step #3, LOAD DATA gets the metadata for the
table before it runs the move statement on the
files. Subsequent queries on the table will not
need to reload metadata and can access the file
quickly after the move.

This changes the ordering to put the files in place
before loading metadata. This may improve the
likelihood that the filesystem is consistent by
the time we read it. Specifically, we now do:
1. Put the files in directory that the table
   will use when it is created.
2. Create table
Neither of these steps load metadata, so the next
query that runs will load metadata.

Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
---
M tests/common/file_utils.py
1 file changed, 24 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/11959/3
-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 2: Code-Review+1

(2 comments)

Had some questions, but lgtm

http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG@27
PS2, Line 27: This may improve the
            : likelihood that the filesystem is consistent by
            : the time we read it.
The docs at https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html says that S3 provides read-after-write consistency for PUTS of new objects, why does it fall to eventual consistency in our case? I suspect it is the listing of directory contents which is eventual consistent only.

Is there a way to ensure stronger consistency to make the tests completely stable? Wouldn't S3Guard http://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/ solve this problem?


http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py
File tests/common/file_utils.py:

http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py@67
PS2, Line 67: command
nit: introducing 'command' variable seems unnecessary, and we don't do that at other places in the file either



-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 11:42:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3482/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 21:45:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 3: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG@27
PS2, Line 27: This may improve the
            : likelihood that the filesystem is consistent by
            : the time we read it.
> Based on the s3 tracing, we always do an existence check before we do a wri
Thanks for the explanation!



-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Nov 2018 13:55:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 4: Code-Review+2

Rebased, carry +2


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Nov 2018 21:54:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11959

to look at the new patch set (#2).

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................

IMPALA-7804: Mitigate s3 consistency issues for test_scanners

test_scanners.py has seen several flaky failures on
s3 due to eventual consistency. The symptom is Impala
being unable to read a file that it just loaded to s3.

A large number of tables used in test_scanners.py
use the file_utils helper functions for creating
the tables. These follow the pattern:
1. Copy files to temporary directory in HDFS/S3/etc
2. Create table
3. Run LOAD DATA to move the files to the table

In step #3, LOAD DATA gets the metadata for the
table before it runs the move statement on the
files. Subsequent queries on the table will not
need to reload metadata and can access the file
quickly after the move.

This changes the ordering to put the files in place
before loading metadata. This may improve the
likelihood that the filesystem is consistent by
the time we read it. Specifically, we now do:
1. Put the files in directory that the table
   will use when it is created.
2. Create table
Neither of these steps load metadata, so the next
query that runs will load metadata.

Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
---
M tests/common/file_utils.py
1 file changed, 25 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/11959/2
-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1413/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 21:40:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................

IMPALA-7804: Mitigate s3 consistency issues for test_scanners

test_scanners.py has seen several flaky failures on
s3 due to eventual consistency. The symptom is Impala
being unable to read a file that it just loaded to s3.

A large number of tables used in test_scanners.py
use the file_utils helper functions for creating
the tables. These follow the pattern:
1. Copy files to temporary directory in HDFS/S3/etc
2. Create table
3. Run LOAD DATA to move the files to the table

In step #3, LOAD DATA gets the metadata for the
table before it runs the move statement on the
files. Subsequent queries on the table will not
need to reload metadata and can access the file
quickly after the move.

This changes the ordering to put the files in place
before loading metadata. This may improve the
likelihood that the filesystem is consistent by
the time we read it. Specifically, we now do:
1. Put the files in directory that the table
   will use when it is created.
2. Create table
Neither of these steps load metadata, so the next
query that runs will load metadata.

Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Reviewed-on: http://gerrit.cloudera.org:8080/11959
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/common/file_utils.py
1 file changed, 24 insertions(+), 13 deletions(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Nov 2018 01:53:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1403/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 02:55:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 3: Code-Review+1

Carry +1


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 21:01:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 2:

I'm running on s3 to see if it helps. This is somewhat experimental.


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 02:08:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 06:12:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3506/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Nov 2018 21:55:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 3:

Ran s3 tests three times and didn't see any failures on test_scanners.py tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 22:09:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1402/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 02:40:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11959/2//COMMIT_MSG@27
PS2, Line 27: This may improve the
            : likelihood that the filesystem is consistent by
            : the time we read it.
> The docs at https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.ht
Based on the s3 tracing, we always do an existence check before we do a write. This puts us in the eventually consistent case. I think one thing we can do if we know the destination won't exist is to get rid of the existence check. That should change us over to read-after-write consistency.

S3 tests are definitely going to need more than this change. I'm interested in trying out S3Guard. I haven't tested it, so I'm not sure whether it fixes this issue. S3Guard looks like it helps for listing directories, but we're actually trying to read the file. We'll see if it helps. There might be further changes to read our writes and wait for consistency.


http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py
File tests/common/file_utils.py:

http://gerrit.cloudera.org:8080/#/c/11959/2/tests/common/file_utils.py@67
PS2, Line 67: command
> nit: introducing 'command' variable seems unnecessary, and we don't do that
You're right, there's no need. I think this came about from an earlier version where I was trying to work around a long line (which is now no longer long). Fixed.



-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 21:01:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7804: Mitigate s3 consistency issues for test scanners

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11959 )

Change subject: IMPALA-7804: Mitigate s3 consistency issues for test_scanners
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3479/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/11959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id042496beabe0d0226b347e0653b820fee369f4e
Gerrit-Change-Number: 11959
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Nov 2018 02:08:53 +0000
Gerrit-HasComments: No