You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Lars Volker (Code Review)" <ge...@cloudera.org> on 2016/10/20 21:33:42 UTC
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Lars Volker has uploaded a new change for review.
http://gerrit.cloudera.org:8080/4776
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
IMPALA-4329: Prevent crash in scheduler when no backends are registered
The scheduler crashed with a segmentation fault when there were no
backends registered. This change adds a check and aborts scheduling with
an error. It also contains a test.
Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
---
M be/src/scheduling/simple-scheduler-test-util.cc
M be/src/scheduling/simple-scheduler-test-util.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
4 files changed, 32 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/4776/1
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 4: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4776/1/be/src/scheduling/simple-scheduler.cc
File be/src/scheduling/simple-scheduler.cc:
Line 589: replica_host.hostname, &backend_ip);
> Thanks for pointing this out, it is indeed not obvious.
Thanks, I missed that there is a second call to SelectBackendOnHost() for remote.
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4776/1/be/src/scheduling/simple-scheduler.cc
File be/src/scheduling/simple-scheduler.cc:
Line 589: replica_host.hostname, &backend_ip);
can you elaborate on how an empty backend_config leads to that crash? It's not obvious, and it looks like in that case we should get has_local_backend==false here, and then take the continue on L627, and not execute the code that crashed. What am I missing?
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Henry Robinson (Code Review)" <ge...@cloudera.org>.
Henry Robinson has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/4776/1/be/src/scheduling/simple-scheduler.cc
File be/src/scheduling/simple-scheduler.cc:
PS1, Line 541:
indentation - did you clang-format?
PS1, Line 541: "
I think you should add an error code here (NO_REGISTERED_BACKENDS).
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Henry Robinson (Code Review)" <ge...@cloudera.org>.
Henry Robinson has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 3: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4776/3//COMMIT_MSG
Commit Message:
PS3, Line 12: the top of assignment_ctx.assignment_heap SelectRemoteBackendHost()
grammar, plus maybe say ", called from ComputeScanRangeAssignment()", because that's where the bug actually is.
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
IMPALA-4329: Prevent crash in scheduler when no backends are registered
The scheduler crashed with a segmentation fault when there were no
backends registered: After not being able to find a local backend (none
are configured at all) in ComputeScanRangeAssignment(), the previous
code would eventually try to return the top of
assignment_ctx.assignment_heap in SelectRemoteBackendHost(), but that
heap would be empty. Subsequently, when using the IP address of that
heap node, a segmentation fault would occur.
This change adds a check and aborts scheduling with an error. It also
contains a test.
Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Reviewed-on: http://gerrit.cloudera.org:8080/4776
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Internal Jenkins
---
M be/src/scheduling/simple-scheduler-test-util.cc
M be/src/scheduling/simple-scheduler-test-util.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M common/thrift/generate_error_codes.py
5 files changed, 35 insertions(+), 6 deletions(-)
Approvals:
Internal Jenkins: Verified
Dan Hecht: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has uploaded a new patch set (#2).
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
IMPALA-4329: Prevent crash in scheduler when no backends are registered
The scheduler crashed with a segmentation fault when there were no
backends registered. This change adds a check and aborts scheduling with
an error. It also contains a test.
Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
---
M be/src/scheduling/simple-scheduler-test-util.cc
M be/src/scheduling/simple-scheduler-test-util.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M common/thrift/generate_error_codes.py
5 files changed, 35 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/4776/2
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Hello Henry Robinson,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/4776
to look at the new patch set (#4).
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
IMPALA-4329: Prevent crash in scheduler when no backends are registered
The scheduler crashed with a segmentation fault when there were no
backends registered: After not being able to find a local backend (none
are configured at all) in ComputeScanRangeAssignment(), the previous
code would eventually try to return the top of
assignment_ctx.assignment_heap in SelectRemoteBackendHost(), but that
heap would be empty. Subsequently, when using the IP address of that
heap node, a segmentation fault would occur.
This change adds a check and aborts scheduling with an error. It also
contains a test.
Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
---
M be/src/scheduling/simple-scheduler-test-util.cc
M be/src/scheduling/simple-scheduler-test-util.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M common/thrift/generate_error_codes.py
5 files changed, 35 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/4776/4
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 1:
(3 comments)
Thanks for the comments. Please see PS3.
http://gerrit.cloudera.org:8080/#/c/4776/1/be/src/scheduling/simple-scheduler.cc
File be/src/scheduling/simple-scheduler.cc:
PS1, Line 541:
> indentation - did you clang-format?
I hadn't, done now (and added a pre-commit hook).
PS1, Line 541: "
> I think you should add an error code here (NO_REGISTERED_BACKENDS).
Done
Line 589: replica_host.hostname, &backend_ip);
> can you elaborate on how an empty backend_config leads to that crash? It's
Thanks for pointing this out, it is indeed not obvious.
After determining a remote read, the code would eventually try to take the top element of assignment_ctx.assignment_heap_ in L975. Subsequently, when using the returned IP it would segfault. I added this to the commit message, too.
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 3:
(1 comment)
Thanks Henry, please see PS4.
http://gerrit.cloudera.org:8080/#/c/4776/3//COMMIT_MSG
Commit Message:
PS3, Line 12: the top of assignment_ctx.assignment_heap SelectRemoteBackendHost()
> grammar, plus maybe say ", called from ComputeScanRangeAssignment()", becau
Done
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4329: Prevent crash in scheduler when no backends are registered
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has uploaded a new patch set (#3).
Change subject: IMPALA-4329: Prevent crash in scheduler when no backends are registered
......................................................................
IMPALA-4329: Prevent crash in scheduler when no backends are registered
The scheduler crashed with a segmentation fault when there were no
backends registered: After not being able to find a local backend (none
are configured at all), the previous code would eventually try to return
the top of assignment_ctx.assignment_heap SelectRemoteBackendHost(),
but that heap would be empty. Subsequently, when using the IP address of
that heap node, a segmentation fault would occur.
This change adds a check and aborts scheduling with
an error. It also contains a test.
Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
---
M be/src/scheduling/simple-scheduler-test-util.cc
M be/src/scheduling/simple-scheduler-test-util.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M common/thrift/generate_error_codes.py
5 files changed, 35 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/4776/3
--
To view, visit http://gerrit.cloudera.org:8080/4776
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d93158f34841ea66dc3682290266262c87ea7ff
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>