You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Lars Volker (Code Review)" <ge...@cloudera.org> on 2016/05/03 15:35:06 UTC
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Lars Volker has uploaded a new change for review.
http://gerrit.cloudera.org:8080/2431
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
IMPALA-2918: Unit test framework for simple scheduler
The SimpleScheduler class is currently hard to change because of the lack of comprehensive
tests. This change adds support classes to make writing tests easier.
The overall testing approach looks like this: Each test builds a list of hosts and a plan,
both to which elements can be added using various helper methods. Then scheduling can be
tested by instantiating SchedulerWrapper and calling Compute(...). The result can be
verified using a set of helper methods. There are also helper methods to modify the
internal state of the scheduler between subsequent calls to SchedulerWrapper::Compute().
The model currently comes with some known limitations:
- Files map 1:1 to blocks and to scan ranges.
- All files have the same size (i.e., 1). Tables that differ in size can be expressed as
having a different number of files.
- We don't support multiple backends on a single host.
- Ports are assigned to hosts automatically and are not configurable by the test.
Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
---
M be/src/scheduling/scheduler.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
4 files changed, 1,141 insertions(+), 192 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/2431/14
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 14
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
Patch Set 16: Code-Review+2
Another rebase. GVM said it was merged, but apparently is wasn't. +2 from Marcel.
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 16
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has uploaded a new patch set (#15).
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
IMPALA-2918: Unit test framework for simple scheduler
The SimpleScheduler class is currently hard to change because of the lack of
comprehensive tests. This change adds support classes to make writing tests
easier.
The overall testing approach looks like this: Each test builds a list of hosts,
a physical schema, and a plan, to all of which elements can be added using
various helper methods. Then scheduling can be tested by instantiating
SchedulerWrapper and calling Compute(...). The result can be verified using a
set of helper methods. There are also helper methods to modify the internal
state of the scheduler between subsequent calls to SchedulerWrapper::Compute().
The model currently comes with some known limitations:
- Files map 1:1 to blocks and to scan ranges.
- All files have the same size (i.e., 1MB). Tables that differ in size can be
expressed as having a different number of files.
- We don't support multiple backends on a single host.
- Ports are assigned to hosts automatically and are not configurable by the
test.
Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
---
M be/src/scheduling/scheduler.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
4 files changed, 1,141 insertions(+), 192 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/2431/15
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 15
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Hello Marcel Kornacker, Matthew Jacobs,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/2431
to look at the new patch set (#15).
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
IMPALA-2918: Unit test framework for simple scheduler
The SimpleScheduler class is currently hard to change because of the lack of
comprehensive tests. This change adds support classes to make writing tests
easier.
The overall testing approach looks like this: Each test builds a list of hosts,
a physical schema, and a plan, to all of which elements can be added using
various helper methods. Then scheduling can be tested by instantiating
SchedulerWrapper and calling Compute(...). The result can be verified using a
set of helper methods. There are also helper methods to modify the internal
state of the scheduler between subsequent calls to SchedulerWrapper::Compute().
The model currently comes with some known limitations:
- Files map 1:1 to blocks and to scan ranges.
- All files have the same size (i.e., 1MB). Tables that differ in size can be
expressed as having a different number of files.
- We don't support multiple backends on a single host.
- Ports are assigned to hosts automatically and are not configurable by the
test.
Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
---
M be/src/scheduling/scheduler.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
4 files changed, 1,141 insertions(+), 192 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/2431/15
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 15
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
IMPALA-2918: Unit test framework for simple scheduler
The SimpleScheduler class is currently hard to change because of the lack of
comprehensive tests. This change adds support classes to make writing tests
easier.
The overall testing approach looks like this: Each test builds a list of hosts,
a physical schema, and a plan, to all of which elements can be added using
various helper methods. Then scheduling can be tested by instantiating
SchedulerWrapper and calling Compute(...). The result can be verified using a
set of helper methods. There are also helper methods to modify the internal
state of the scheduler between subsequent calls to SchedulerWrapper::Compute().
The model currently comes with some known limitations:
- Files map 1:1 to blocks and to scan ranges.
- All files have the same size (i.e., 1MB). Tables that differ in size can be
expressed as having a different number of files.
- We don't support multiple backends on a single host.
- Ports are assigned to hosts automatically and are not configurable by the
test.
Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Reviewed-on: http://gerrit.cloudera.org:8080/2431
Reviewed-by: Lars Volker <lv...@cloudera.com>
Tested-by: Internal Jenkins
---
M be/src/scheduling/scheduler.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
4 files changed, 1,141 insertions(+), 192 deletions(-)
Approvals:
Lars Volker: Looks good to me, approved
Internal Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 17
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
Patch Set 15: Code-Review+2
+2 from Marcel, rebased, re-formatted commit msg.
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 15
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-2918: Unit test framework for simple scheduler
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Hello Marcel Kornacker, Matthew Jacobs,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/2431
to look at the new patch set (#16).
Change subject: IMPALA-2918: Unit test framework for simple scheduler
......................................................................
IMPALA-2918: Unit test framework for simple scheduler
The SimpleScheduler class is currently hard to change because of the lack of
comprehensive tests. This change adds support classes to make writing tests
easier.
The overall testing approach looks like this: Each test builds a list of hosts,
a physical schema, and a plan, to all of which elements can be added using
various helper methods. Then scheduling can be tested by instantiating
SchedulerWrapper and calling Compute(...). The result can be verified using a
set of helper methods. There are also helper methods to modify the internal
state of the scheduler between subsequent calls to SchedulerWrapper::Compute().
The model currently comes with some known limitations:
- Files map 1:1 to blocks and to scan ranges.
- All files have the same size (i.e., 1MB). Tables that differ in size can be
expressed as having a different number of files.
- We don't support multiple backends on a single host.
- Ports are assigned to hosts automatically and are not configurable by the
test.
Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
---
M be/src/scheduling/scheduler.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
4 files changed, 1,141 insertions(+), 192 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/2431/16
--
To view, visit http://gerrit.cloudera.org:8080/2431
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7aee9b16067e8728a8e96d4def3e568ad21f4bf
Gerrit-PatchSet: 16
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Silvius Rus <sr...@cloudera.com>