You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2021/01/14 17:40:11 UTC

[Impala-ASF-CR] WIP IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16949


Change subject: WIP IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

WIP IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug action to simulate disk write error and adds new
test cases.

Testing:
 - TODO: add new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
10 files changed, 113 insertions(+), 9 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/1
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8051/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 13
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 Jan 2021 02:50:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8015/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Jan 2021 23:15:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................

IMPALA-9224: Blacklist nodes with faulty disk for spilling

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also classifies disk error codes and defines a blacklistable error set
for non-transient disk errors. Coordinator blacklists executor only if
the executor hitted blacklistable error during spill-to-disk.

Adds a new debug action to simulate disk write error during spill-to-
disk. To use, specify in query options as:
  'debug_action': 'IMPALA_TMP_FILE_WRITE:<hostname>:<port>:<action>'

  where <hostname> and <port> represent the impalad which execute the
  fragment instances, <port> is the BE krpc port (default 27000).

Adds new test cases for blacklist and query-retry to cover the code
changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.
 - TODO: simulate disk failures with tools on a cluster.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/error-converter.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
14 files changed, 510 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/14
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 14
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 16: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 16
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Feb 2021 05:12:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8016/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jan 2021 07:01:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8015/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jan 2021 06:50:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: WIP IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8005/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Jan 2021 18:01:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/16949/2/tests/custom_cluster/test_blacklist.py
File tests/custom_cluster/test_blacklist.py:

http://gerrit.cloudera.org:8080/#/c/16949/2/tests/custom_cluster/test_blacklist.py@189
PS2, Line 189: @SkipIfNotHdfsMinicluster.tuned_for_minicluster
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/16949/2/tests/custom_cluster/test_query_retries.py
File tests/custom_cluster/test_query_retries.py:

http://gerrit.cloudera.org:8080/#/c/16949/2/tests/custom_cluster/test_query_retries.py@48
PS2, Line 48: def _get_disk_fail_action(port):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/16949/2/tests/custom_cluster/test_query_retries.py@1004
PS2, Line 1004: @SkipIfNotHdfsMinicluster.tuned_for_minicluster
flake8: E302 expected 2 blank lines, found 1



-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jan 2021 06:31:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 12:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/io/error-converter.cc
File be/src/runtime/io/error-converter.cc:

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/io/error-converter.cc@62
PS12, Line 62:   static const set<int32_t> non_blacklistable_disk_error_codes = {
> I think its safer to enumerate out the errnos that we want to blacklist for
Right, we should be more conservative. Will fix it.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/query-state.cc@71
PS12, Line 71: DEFINE_bool(reporting_disk_health_enabled, false,
> I know I was the one who told you to add this, but I realized we already ha
Will remove it.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/tmp-file-mgr.cc@679
PS12, Line 679:     if (ErrorConverter::IsBlacklistableError(status)) {
> So we're not considering 'num_blacklisted_files_' here. I guess the point i
Will remove blacklisting for single reading error from this patch. I was thinking to change it in more conservative way in following patch, like adding stats in TmpFileMgr class to count disk io failures in last 10 minutes and blacklisting node if the count exceed a certain threshold.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/tmp-file-mgr.cc@797
PS12, Line 797:           if (ErrorConverter::IsBlacklistableError(err)) {
> I think the logic here is that if a single one of the errors that we got wa
Agree, will fix it.


http://gerrit.cloudera.org:8080/#/c/16949/12/tests/custom_cluster/test_blacklist.py
File tests/custom_cluster/test_blacklist.py:

http://gerrit.cloudera.org:8080/#/c/16949/12/tests/custom_cluster/test_blacklist.py@326
PS12, Line 326:     assert results.success
> I think you should be able to check that the runtime profile contains the "
Ok, will add code to check runtime profile.


http://gerrit.cloudera.org:8080/#/c/16949/12/tests/custom_cluster/test_blacklist.py@328
PS12, Line 328:     client.close_query(handle)
> Maybe add a sleep that's long enough to see the BE get removed from the bla
Will add another query after blacklisting timeout.



-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Jan 2021 20:32:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8046/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Jan 2021 19:08:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 15: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 15
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Feb 2021 23:26:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8028/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Jan 2021 01:32:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug action to simulate disk write error and adds new test
cases for blacklist and query-retry.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 400 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/4
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................

IMPALA-9224: Blacklist nodes with faulty disk for spilling

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also classifies disk error codes and defines a blacklistable error set
for non-transient disk errors. Coordinator blacklists executor only if
the executor hitted blacklistable error during spill-to-disk.

Adds debug actions to simulate disk write/read errors and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.
 - TODO: simulate disk failures with tools on a cluster.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/error-converter.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
14 files changed, 580 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/11
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 6:

(3 comments)

As discussed, it would be good to add a startup flag for this - probably default to having it off for now until we've done some of the planned follow up tasks, and then when we eventually switch it to always on it still allows users who run into problems with it to disable it.

http://gerrit.cloudera.org:8080/#/c/16949/6/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:

http://gerrit.cloudera.org:8080/#/c/16949/6/be/src/runtime/query-state.cc@347
PS6, Line 347: #ifndef NDEBUG
I think this should be removed - its doesn't really help much from a performance perspective, since InitBufferPoolState is only called once per backend per query and checking to see if the string is empty is fast, and it will mean that the tests you wrote will fail in release build jobs, which is not what we want.

It does at least protect against users accidentally setting a --debug_action in production environments, but we just rely on users not making that mistake.


http://gerrit.cloudera.org:8080/#/c/16949/6/common/protobuf/control_service.proto
File common/protobuf/control_service.proto:

http://gerrit.cloudera.org:8080/#/c/16949/6/common/protobuf/control_service.proto@263
PS6, Line 263: runs
nit: 'ran'


http://gerrit.cloudera.org:8080/#/c/16949/6/common/protobuf/control_service.proto@265
PS6, Line 265: spilling_local_faulty_disk
I think this is unnecessarily verbose, how about just 'local_disk_faulty' or 'spilling_disk_faulty'? (and of course, make the other places like function names match).



-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Jan 2021 22:09:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8030/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Jan 2021 06:53:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug action to simulate disk write error and adds new
test cases for blacklist and query-retry.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 399 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/2
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8020/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Jan 2021 18:22:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8037/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Jan 2021 17:40:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................

IMPALA-9224: Blacklist nodes with faulty disk for spilling

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also classifies disk error codes and defines a blacklistable error set
for non-transient disk errors. Coordinator blacklists executor only if
the executor hitted blacklistable error during spill-to-disk.

Adds a new debug action to simulate disk write error during spill-to-
disk. To use, specify in query options as:
  'debug_action': 'IMPALA_TMP_FILE_WRITE:<hostname>:<port>:<action>'

  where <hostname> and <port> represent the impalad which execute the
  fragment instances, <port> is the BE krpc port (default 27000).

Adds new test cases for blacklist and query-retry to cover the code
changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.
 - Manually simulated disk failures in scratch directories on nodes
   of a cluster, verified that the nodes were blacklisted as
   expected.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Reviewed-on: http://gerrit.cloudera.org:8080/16949
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/error-converter.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
14 files changed, 510 insertions(+), 11 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 17
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................

IMPALA-9224: Blacklist nodes with faulty disk for spilling

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also classifies disk error codes and defines a blacklistable error set
for non-transient disk errors. Coordinator blacklists executor only if
the executor hitted blacklistable error during spill-to-disk.

Adds debug actions to simulate disk write error and adds new test
cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.
 - TODO: simulate disk failures with tools on a cluster.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/error-converter.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
14 files changed, 510 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/13
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 13
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug actions to simulate disk write/read errors and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 506 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/9
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................

IMPALA-9224: Blacklist nodes with faulty disk for spilling

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also classifies disk error codes and defines a blacklistable error set
for non-transient disk errors. Coordinator blacklists executor only if
the executor hitted blacklistable error during spill-to-disk.

Adds a new debug action to simulate disk write error during spill-to-
disk. To use, specify in query options as:
  'debug_action': 'IMPALA_TMP_FILE_WRITE:<hostname>:<port>:<action>'

  where <hostname> and <port> represent the impalad which execute the
  fragment instances, <port> is the BE krpc port (default 27000).

Adds new test cases for blacklist and query-retry to cover the code
changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.
 - Manually simulated disk failures in scratch directories on nodes
   of a cluster, verified that the nodes were blacklisted as
   expected.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/error-converter.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
14 files changed, 510 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/15
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 15
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6869/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 16
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Feb 2021 23:26:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug actions to simulate disk write/read errors and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 506 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/8
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug action to simulate disk write error and adds new test
cases for blacklist and query-retry.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 402 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/3
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8033/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Jan 2021 23:26:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug actions to simulate disk write/read errors and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 506 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/10
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 15
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Feb 2021 18:15:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................


Patch Set 6:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/16949/6/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:

http://gerrit.cloudera.org:8080/#/c/16949/6/be/src/runtime/query-state.cc@347
PS6, Line 347: #ifndef NDEBUG
> I think this should be removed - its doesn't really help much from a perfor
Removed.


http://gerrit.cloudera.org:8080/#/c/16949/6/common/protobuf/control_service.proto
File common/protobuf/control_service.proto:

http://gerrit.cloudera.org:8080/#/c/16949/6/common/protobuf/control_service.proto@263
PS6, Line 263: runs
> nit: 'ran'
Fixed as suggested.


http://gerrit.cloudera.org:8080/#/c/16949/6/common/protobuf/control_service.proto@265
PS6, Line 265: spilling_local_faulty_disk
> I think this is unnecessarily verbose, how about just 'local_disk_faulty' o
Renamed as local_disk_faulty.



-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Jan 2021 01:04:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 13:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/16949/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/16949/9//COMMIT_MSG@7
PS9, Line 7:  Blackli
> This patch seems pretty self-contained, so I'm not sure there needs to be a
removed part 1.


http://gerrit.cloudera.org:8080/#/c/16949/9//COMMIT_MSG@13
PS9, Line 13: the executor hitted blacklistable error during spill-to-disk.
> It would be good to add a brief description of the flag that you're adding 
Added description for new debug action.



-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 13
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 Jan 2021 02:44:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8053/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 14
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 Jan 2021 03:08:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 12:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/16949/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/16949/9//COMMIT_MSG@7
PS9, Line 7:  Blackli
This patch seems pretty self-contained, so I'm not sure there needs to be a "part 2", vs. filing separate JIRAs for the other tasks we have planned, but if there is something that you think should be included as a "part 2" could you briefly describe what that is in the message below.


http://gerrit.cloudera.org:8080/#/c/16949/9//COMMIT_MSG@13
PS9, Line 13: the executor hitted blacklistable error during spill-to-disk.
It would be good to add a brief description of the flag that you're adding here.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/io/error-converter.cc
File be/src/runtime/io/error-converter.cc:

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/io/error-converter.cc@62
PS12, Line 62:   static const set<int32_t> non_blacklistable_disk_error_codes = {
I think its safer to enumerate out the errnos that we want to blacklist for, rather than the ones we don't want to blacklist for, so that the default behavior is more conservative - eg. if a new error code got added by the underlying system such that we don't know about it, the default behavior should be to not blacklist for that code.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/query-state.cc@71
PS12, Line 71: DEFINE_bool(reporting_disk_health_enabled, false,
I know I was the one who told you to add this, but I realized we already have a flag to turn this off if problems occur (--blacklisting_enabled) and we probably don't need to have separate flags for individually turning on and off each blacklisting scenario.

Plus, with the new error code stuff I feel more comfortable that this patch isn't going to have unintended consequences.

If you would still prefer to keep it for now, I would recommend marking it as "experimental" rather than "advanced" as that gives us more room to remove it in the future.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:

http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/tmp-file-mgr.cc@679
PS12, Line 679:     if (ErrorConverter::IsBlacklistableError(status)) {
So we're not considering 'num_blacklisted_files_' here. I guess the point is that for writes a single failure doesn't necessarily fail the query, since we can try to write to the other files, but for reads a single failure does fail the overall query, cause we only wrote this once, so there's no where else to try reading it from?

I'm concerned that may make us a bit too aggressive about blacklisting, so maybe we should only do the blacklisting on the write path? After all, if a read fails because the disk really is completely faulty, we'll figure that out on the next query that spills when it can't do any writing.


http://gerrit.cloudera.org:8080/#/c/16949/12/be/src/runtime/tmp-file-mgr.cc@797
PS12, Line 797:           if (ErrorConverter::IsBlacklistableError(err)) {
I think the logic here is that if a single one of the errors that we got was blacklistable then we blacklist it, but do we maybe want it to be that we require that all errors are blacklistable before blacklisting.

If any one of the errors was recoverable (i.e. not blacklistable) then seems like our expectation is that future queries really could succeed.


http://gerrit.cloudera.org:8080/#/c/16949/12/tests/custom_cluster/test_blacklist.py
File tests/custom_cluster/test_blacklist.py:

http://gerrit.cloudera.org:8080/#/c/16949/12/tests/custom_cluster/test_blacklist.py@326
PS12, Line 326:     assert results.success
I think you should be able to check that the runtime profile contains the "Blacklisted Executors" line with the corresponding backend.


http://gerrit.cloudera.org:8080/#/c/16949/12/tests/custom_cluster/test_blacklist.py@328
PS12, Line 328:     client.close_query(handle)
Maybe add a sleep that's long enough to see the BE get removed from the blacklist and then run another query that doesn't have the debug action and show that the profile shows it running on all 3 backends.



-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Jan 2021 19:11:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug actions to simulate disk write/read errors and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 506 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/7
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................

IMPALA-9224: Blacklist nodes with faulty disk for spilling

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also classifies disk error codes and defines a non-blacklistable error
set for transient disk errors. Coordinator blacklists executor only if
the executor hitted blacklistable error during spill-to-disk.

Adds debug actions to simulate disk write/read errors and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.
 - TODO: simulate disk failures with tools on a cluster.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/error-converter.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
14 files changed, 572 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/12
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug action to simulate disk write error and adds new test
cases for blacklist and query-retry.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 398 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/5
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8041/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Jan 2021 22:16:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224 (part 1): Blacklist nodes with faulty disks

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224 (part 1): Blacklist nodes with faulty disks
......................................................................

IMPALA-9224 (part 1): Blacklist nodes with faulty disks

This patch extends blacklist functionality by adding executor node to
blacklist if a query fails caused by disk failure during spill-to-disk.
Also adds debug action to simulate disk write/read error and adds new
test cases for blacklist and query-retry to cover the code changes.

Testing:
 - Passed new test cases.
 - Passed exhaustive test.

Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/query-state.cc
M be/src/runtime/tmp-file-mgr-internal.h
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/protobuf/control_service.proto
M common/thrift/generate_error_codes.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_query_retries.py
12 files changed, 502 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16949/6
-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8042/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Jan 2021 22:31:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9224: Blacklist nodes with faulty disk for spilling

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16949 )

Change subject: IMPALA-9224: Blacklist nodes with faulty disk for spilling
......................................................................


Patch Set 16: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/16949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04bfcb7f2e0b1ef24a5b4350f270feecd8c47437
Gerrit-Change-Number: 16949
Gerrit-PatchSet: 16
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Feb 2021 23:26:39 +0000
Gerrit-HasComments: No