You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2018/12/14 18:08:16 UTC

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12088

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................

Improve SCRATCH_ALLOCATION_FAILED error

Adds additional context about how much scratch was allocated
by the query and the impalad in total. We sometimes see scratch
allocation failures because a query was spilling heavily and
ate up all the disk. In this case, the high values in the
error should provide an additional clue that the volume
of spilling is the problem (rather than disks being full
for other reasons).

Example error after deleting /tmp/impala-scratch:

[localhost:21000] default> set mem_limit=150m; select distinct * from tpch_parquet.lineitem limit 5;
WARNINGS: Could not create files in any configured scratch directories (--scratch_dirs=/tmp/impala-scratch) on backend 'tarmstrong-box:22000'. 2.00 MB of scratch is currently in use by Impala on this backend (2.00 MB by this query). See logs for previous errors that may have prevented creating or writing scratch files.
Disk I/O error: open() failed for /tmp/impala-scratch/7d473ea7aef26431:c9105f7900000000_3120108e-475b-4616-9825-8bbdb1dc9cc2. The given path doesn't exist. errno=2

Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/thrift/generate_error_codes.py
3 files changed, 21 insertions(+), 13 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/12088/1
--
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3571/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 23:11:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 23:11:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.

Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 2: Code-Review+2

> tmp-file-mgr-test exercises the code paths. It checks the error
 > code only, not the full message (which is why I didn't have to
 > update it).

Thanks for the clarification.


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 23:03:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................

Improve SCRATCH_ALLOCATION_FAILED error

Adds additional context about how much scratch was allocated
by the query and the impalad in total. We sometimes see scratch
allocation failures because a query was spilling heavily and
ate up all the disk. In this case, the high values in the
error should provide an additional clue that the volume
of spilling is the problem (rather than disks being full
for other reasons).

Example error after deleting /tmp/impala-scratch:

[localhost:21000] default> set mem_limit=150m; select distinct * from tpch_parquet.lineitem limit 5;
WARNINGS: Could not create files in any configured scratch directories (--scratch_dirs=/tmp/impala-scratch) on backend 'tarmstrong-box:22000'. 2.00 MB of scratch is currently in use by this Impala Daemon (2.00 MB by this query). See logs for previous errors that may have prevented creating or writing scratch files.
Disk I/O error: open() failed for /tmp/impala-scratch/7d473ea7aef26431:c9105f7900000000_3120108e-475b-4616-9825-8bbdb1dc9cc2. The given path doesn't exist. errno=2

Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Reviewed-on: http://gerrit.cloudera.org:8080/12088
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/runtime/tmp-file-mgr.h
M common/thrift/generate_error_codes.py
3 files changed, 21 insertions(+), 13 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.

Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 2:

tmp-file-mgr-test exercises the code paths. It checks the error code only, not the full message (which is why I didn't have to update it).


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 23:02:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1607/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 18:50:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.

Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................

Improve SCRATCH_ALLOCATION_FAILED error

Example error after deleting /tmp/impala-scratch:

[localhost:21000] default> set mem_limit=150m; select distinct * from tpch_parquet.lineitem limit 5;
WARNINGS: Could not create files in any configured scratch directories (--scratch_dirs=/tmp/impala-scratch) on backend 'tarmstrong-box:22000'. 2.00 MB of scratch is currently in use by this Impala Daemon (2.00 MB by this query). See logs for previous errors that may have prevented creating or writing scratch files.
Disk I/O error: open() failed for /tmp/impala-scratch/7d473ea7aef26431:c9105f7900000000_3120108e-475b-4616-9825-8bbdb1dc9cc2. The given path doesn't exist. errno=2

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/12088/2
--
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.

Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 2:

Do we have tests for these code paths?


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 19:27:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 15 Dec 2018 03:07:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Improve SCRATCH ALLOCATION FAILED error

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12088 )

Change subject: Improve SCRATCH_ALLOCATION_FAILED error
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1606/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12088
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbedd586c57ec02e784143927e82b74455f98dc8
Gerrit-Change-Number: 12088
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Dec 2018 18:42:13 +0000
Gerrit-HasComments: No