You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2016/12/22 06:58:37 UTC
[kudu-CR] [jenkins] added JEPSEN build configuration
Alexey Serbin has uploaded a new change for review.
http://gerrit.cloudera.org:8080/5559
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
[jenkins] added JEPSEN build configuration
Added a new build type to the build-and-test.sh jenkins script.
This is to run the kudu.jepsen consistency test along with DEBUG,
RELEASE, LINT and other build configurations.
Currently, the set of nodes to run Kudu cluster should be provisioned
prior to running the test. The names (or IP addresses) of those nodes
should be specified via environment variables.
The test uses a Jenkins slave as the Jepsen control node, running the
control logic and the Kudu client (Java) on the machine. All necessary
operations on the Kudu cluster nodes and run via SSH. The private SSH
key should be set accordingly: it should be either added into the SSH
agent's key ring on the control node or the path to the key specified
via corresponding environment variable. In the latter case the key
must be stored in plain/non-encrypted form.
Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
---
M build-support/jenkins/build-and-test.sh
1 file changed, 81 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/5559/1
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
Patch Set 3:
> As Todd mentioned, the upstream Kudu Jenkins instance probably
> can't communicate with internal Cloudera resources, so even if
> build-and-test.sh was configured with the new parameters correctly,
> it likely won't work. Plus, it's probably overkill to run Jepsen
> precommit anyway.
>
> I think you've pretty much abandoned this in favor of a dedicated
> Cloudera-internal jepsen nightly test, right?
This script is used to run http://sandbox.jenkins.cloudera.com/view/Kudu/job/kudu-jepsen/
I can separate this into another file, if needed.
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
Patch Set 3:
> > I think you've pretty much abandoned this in favor of a dedicated
> > Cloudera-internal jepsen nightly test, right?
>
> This script is used to run http://sandbox.jenkins.cloudera.com/view/Kudu/job/kudu-jepsen/
>
> I can separate this into another file, if needed.
Oh I see. Do you think this approach is cleaner than a standalone script a la benchmarks.sh or tpch.sh?
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
Patch Set 3:
> > > I think you've pretty much abandoned this in favor of a
> dedicated
> > > Cloudera-internal jepsen nightly test, right?
> >
> > This script is used to run http://sandbox.jenkins.cloudera.com/view/Kudu/job/kudu-jepsen/
> >
> > I can separate this into another file, if needed.
>
> Oh I see. Do you think this approach is cleaner than a standalone
> script a la benchmarks.sh or tpch.sh?
I don't think it's cleaner. I just used it because I initially started it as an update for the jenkins build script. By the time I submitted this patch, I still had some wild ideas that with additional provisions to create virtual machines from the script (like those we use for dist_test), we could make it run as a pre-commit check (that was the 'ideal' target we initially discussed with David).
Now I understand that the consensus is that this test is more like a system test which should not be run with every commit. That's said, I think the approach like benchmarks.sh or tpch.sh is a better/cleaner fit here. I'll update this patch and the Jenkins job at http://sandbox.jenkins.cloudera.com/view/Kudu/job/kudu-jepsen/ accordingly.
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has abandoned this change.
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
Abandoned
Abandoned in favor of http://gerrit.cloudera.org:8080/5624
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
Patch Set 3: Code-Review-1
As Todd mentioned, the upstream Kudu Jenkins instance probably can't communicate with internal Cloudera resources, so even if build-and-test.sh was configured with the new parameters correctly, it likely won't work. Plus, it's probably overkill to run Jepsen precommit anyway.
I think you've pretty much abandoned this in favor of a dedicated Cloudera-internal jepsen nightly test, right?
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5559
to look at the new patch set (#2).
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
[jenkins] added JEPSEN build configuration
Added a new build type to the build-and-test.sh Jenkins script.
This is to run the kudu.jepsen consistency test along with DEBUG,
RELEASE, LINT and other build configurations.
Currently, the set of nodes to run Kudu cluster should be provisioned
prior to running the test. The names (or IP addresses) of those nodes
should be specified via environment variables: first is for the list
of master nodes and another is for the list of tablet server nodes.
The test uses a Jenkins slave as the Jepsen control node, running the
control logic and the Kudu Java client freshly built on the machine.
All control operations on the Kudu cluster nodes are run via SSH.
The private SSH key should be set accordingly: it should be either
added into the SSH agent's key ring on the control node or the path
to the key specified via corresponding environment variable. In the
latter case the key must be stored in plain/non-encrypted form.
Besides, the public part of the SSH key should be added into the
authorized_keys file for the root user on all Kudu cluster machines.
The Kudu cluster should consist of the Linux machines of the same
architecture and OS distro as the Jenkins slaves: the test copies and
runs the newly built Kudu binaries of the server-side components and the
kudu CLI utility on all Kudu cluster nodes.
As for now, the kudu-jepsen test supports recent Debian distros only.
This limitation is not a problem for our current Jenkins setup.
Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
---
M build-support/jenkins/build-and-test.sh
1 file changed, 81 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/5559/2
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [jenkins] added JEPSEN build configuration
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5559
to look at the new patch set (#3).
Change subject: [jenkins] added JEPSEN build configuration
......................................................................
[jenkins] added JEPSEN build configuration
Added a new build type to the build-and-test.sh Jenkins script.
This is to run the kudu.jepsen consistency test along with DEBUG,
RELEASE, LINT and other build configurations.
Currently, the set of nodes to run Kudu cluster should be provisioned
prior to running the test. The names (or IP addresses) of those nodes
should be specified via environment variables: one is for the list
of master nodes and another is for the list of tablet server nodes.
The test uses a Jenkins slave as the Jepsen control node, running the
control logic and the Kudu Java client freshly built on the machine.
All control operations on the Kudu cluster nodes are run via SSH.
The private SSH key should be set accordingly: it should be either
added into the SSH agent's key ring on the control node or the path
to the key specified via corresponding environment variable. In the
latter case the key must be stored in plain/non-encrypted form.
Besides, the public part of the SSH key should be added into the
authorized_keys file for the root user on all Kudu cluster machines.
The Kudu cluster should consist of the Linux machines of the same
architecture and OS distro as the Jenkins slaves: the test copies and
runs the newly built Kudu binaries of the server-side components and the
kudu CLI utility on all Kudu cluster nodes.
As for now, the kudu-jepsen test supports recent Debian distros only.
This limitation is not a problem for our current Jenkins setup.
Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
---
M build-support/jenkins/build-and-test.sh
1 file changed, 80 insertions(+), 19 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/5559/3
--
To view, visit http://gerrit.cloudera.org:8080/5559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia2576717dc8fff3fe2c275f29c2c06eb0d35bddc
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>