You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Philip Zeyliger (Code Review)" <ge...@cloudera.org> on 2017/08/24 21:48:11 UTC

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for scratch failures. 

Philip Zeyliger has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7816

Change subject: IMPALA-3642: Adding backend addresses to error statuses for scratch failures.
......................................................................

IMPALA-3642: Adding backend addresses to error statuses for scratch failures.

Adds GetBackendAddress() (which is host:port) to status messages stemming
from IO failures writing to local scratch disks.

I tested this manually by running a query which spills after
"chmod 000 /tmp/impala-scratch" for one type of error. To wit:

    [hostname:21000] > select count(*) from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey;
    Query: select count(*) from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
    Query submitted at: 2017-08-24 08:48:23 (Coordinator: http://hostname:25000)
    Query progress can be monitored at: http://hostname:25000/query_plan?query_id=5243a68eb540c075:85fd2ccc00000000
    WARNINGS: Could not create files in any configured scratch directories (--scratch_dirs). See logs for previous errors that may have prevented creating or writing scratch files.
    Opening '/tmp/impala-scratch/5243a68eb540c075:85fd2ccc00000000_bbee0878-805f-4881-bf0d-f18c7d18ee83' for write failed with errno=13 description=Error(13): Permission denied backend=hostname:22002
    Opening '/tmp/impala-scratch/5243a68eb540c075:85fd2ccc00000000_bbee0878-805f-4881-bf0d-f18c7d18ee83' for write failed with errno=13 description=Error(13): Permission denied backend=hostname:22002

Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
---
M be/src/runtime/disk-io-mgr.cc
1 file changed, 9 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/7816/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has uploaded a new patch set (#2).

Change subject: IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.
......................................................................

IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Adds GetBackendAddress() (which is host:port) to error messages stemming
from SCRATCH_LIMIT_EXCEEDED, SCRATCH_READ_TRUNCATED, and
SCRATCH_ALLOCATION_FAILED messages.

Testing:

* Unit tests assert the string is updated for SCRATCH_LIMIT_EXCEEDED
  and SCRATCH_ALLOCATION_FAILED. SCRATCH_READ_TRUNCATED doesn't
  have an existing test, and I didn't add a new one.

* Manually testing a query that spills after "chmod 000 /tmp/impala-scratch":
    $ chmod 000 /tmp/impala-scratch
    $ impala-shell
    [dev:21000] > set mem_limit=100m;
    MEM_LIMIT set to 100m
    [dev:21000] > select count(*) from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey;
    Query: select count(*) from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
    Query submitted at: 2017-09-11 11:07:06 (Coordinator: http://dev:25000)
    Query progress can be monitored at: http://dev:25000/query_plan?query_id=5c48ff8f4103c194:1b40a6c00000000
    WARNINGS: Could not create files in any configured scratch directories (--scratch_dirs=/tmp/impala-scratch) on backend 'dev:22002'. See logs for previous errors that may have prevented creating or writing scratch files.
    Opening '/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c00000000_08e8d63b-169d-4571-a0fe-c48fa08d73e6' for write failed with errno=13 description=Error(13): Permission denied
    Opening '/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c00000000_08e8d63b-169d-4571-a0fe-c48fa08d73e6' for write failed with errno=13 description=Error(13): Permission denied
    Opening '/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c00000000_08e8d63b-169d-4571-a0fe-c48fa08d73e6' for write failed with errno=13 description=Error(13): Permission denied

Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M common/thrift/generate_error_codes.py
4 files changed, 13 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/7816/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.
......................................................................


Patch Set 3: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.
......................................................................


Patch Set 2: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for scratch failures.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for scratch failures.
......................................................................


Patch Set 1:

I'm looking for feedback on whether we have more explicit ways to test this sort of thing. This only tackles one particular source of errors.

There are ~294 matches for Status.*Substitute, and of these some (like the runtime errors in filesystem-util.cc) could compellingly make a case for similar treatment. The way Statuses are propagated down to the query is not trivial, though one could add a "source host" field to the relevant Thrift structure and populate it for non-OK statuses in Status::ToThrift(). (Or, even just append it to any messages in an unstructured way.)

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for scratch failures.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for scratch failures.
......................................................................


Patch Set 1:

We have test coverage for scratch errors in a few places:

  be/src/runtime/bufferpool/buffer-pool-test.cc
  be/src/runtime/tmp-file-mgr-test.cc
  tests/custom_cluster/test_scratch_disk.py

There are a bunch of write error tests and a single read error test: BufferPoolTest::ScratchReadError

The backend unit tests check the error codes but not the error messages - one option is to extend them to check the error messages.

I haven't fully thought through how it would work out, but it might be easier to add this information to the error message in be/src/tmp-file-mgr.cc.  The write errors get rolled up into a SCRATCH_ALLOCATION_FAILED message (there may be more than one per disk). There are also a couple of other error types that emanate from that file: SCRATCH_LIMIT_EXCEEDED and SCRATCH_READ_TRUNCATED.

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.
......................................................................


IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Adds GetBackendAddress() (which is host:port) to error messages stemming
from SCRATCH_LIMIT_EXCEEDED, SCRATCH_READ_TRUNCATED, and
SCRATCH_ALLOCATION_FAILED messages.

Testing:

* Unit tests assert the string is updated for SCRATCH_LIMIT_EXCEEDED
  and SCRATCH_ALLOCATION_FAILED. SCRATCH_READ_TRUNCATED doesn't
  have an existing test, and I didn't add a new one.

* Manually testing a query that spills after "chmod 000 /tmp/impala-scratch":
    $ chmod 000 /tmp/impala-scratch
    $ impala-shell
    [dev:21000] > set mem_limit=100m;
    MEM_LIMIT set to 100m
    [dev:21000] > select count(*) from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey;
    Query: select count(*) from tpch_parquet.lineitem join tpch_parquet.orders on l_orderkey = o_orderkey
    Query submitted at: 2017-09-11 11:07:06 (Coordinator: http://dev:25000)
    Query progress can be monitored at: http://dev:25000/query_plan?query_id=5c48ff8f4103c194:1b40a6c00000000
    WARNINGS: Could not create files in any configured scratch directories (--scratch_dirs=/tmp/impala-scratch) on backend 'dev:22002'. See logs for previous errors that may have prevented creating or writing scratch files.
    Opening '/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c00000000_08e8d63b-169d-4571-a0fe-c48fa08d73e6' for write failed with errno=13 description=Error(13): Permission denied
    Opening '/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c00000000_08e8d63b-169d-4571-a0fe-c48fa08d73e6' for write failed with errno=13 description=Error(13): Permission denied
    Opening '/tmp/impala-scratch/5c48ff8f4103c194:1b40a6c00000000_08e8d63b-169d-4571-a0fe-c48fa08d73e6' for write failed with errno=13 description=Error(13): Permission denied

Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Reviewed-on: http://gerrit.cloudera.org:8080/7816
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/runtime/tmp-file-mgr-test.cc
M be/src/runtime/tmp-file-mgr.cc
M common/thrift/generate_error_codes.py
4 files changed, 13 insertions(+), 9 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Tim Armstrong: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.
......................................................................


Patch Set 3: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-3642: Adding backend addresses to error statuses for some scratch failures.
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1212/

-- 
To view, visit http://gerrit.cloudera.org:8080/7816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: If31a50fdf6031312d0348d48aeb8f9688274cac2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No