You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Cassandra Targett <ca...@gmail.com> on 2019/07/01 15:27:16 UTC
Re: [JENKINS] Solr-reference-guide-8.x - Build # 3944 - Still
Failing
I pushed a change to master and updated the Ref Guide master build to use “git-websites” again. I’ll monitor the next time it runs (~30 minutes from now) to see what happens. If it works, I’ll backport the change to branch_8x.
Cassandra
On Jun 30, 2019, 3:13 PM -0500, Erick Erickson <er...@gmail.com>, wrote:
> AFAIC, just push it. Even if you do find a way to test it, we’d still potentially have something _else_ different between your test bed and websites2…..
>
> > On Jun 30, 2019, at 12:46 PM, Cassandra Targett <ca...@gmail.com> wrote:
> >
> > Thanks for filing those issues, Steve.
> >
> > The master branch Ref Guide build started failing this weekend with the Ruby installation problem, so I pinned that build to “websites1” for now.
> >
> > I saw the comments in the INFRA issue about using a newer version of Ruby, but wasn’t really sure how to test installing Ruby 2.5.1 instead. Apparently Gavin tried it and it worked all right after he initialized with the ant ivy-bootstrap target. Should I just commit the version change to the script, or try to test it some more first (somehow)?
> >
> > Cassandra
> > On Jun 27, 2019, 10:41 AM -0500, Steve Rowe <sa...@gmail.com>, wrote:
> > > INFRA JIRA for "websites2" problem: https://issues.apache.org/jira/browse/INFRA-18666
> > >
> > > JIRA to add GPG key download to the ref guide build script: https://issues.apache.org/jira/browse/SOLR-13582
> > >
> > > --
> > > Steve
> > >
> > > > On Jun 27, 2019, at 10:50 AM, Steve Rowe <sa...@gmail.com> wrote:
> > > >
> > > > The 8.x ref guide builds are running on a node they've never run on before: "websites2". The master ref guide builds are running on the "websites1" node, which has not exhibited the same build problems. Both nodes are pinned to label "git-websites", which can result in builds on either the "websites1" node or the "websites2" node. AFAICT there's some kind of routing based on the identity (name maybe?) of the job, which results in effectively permanent assignment to either one or the other node.
> > > >
> > > > I put the key download commands from the error message into the 8.x job config and triggered a run. This enabled verification. But there is now another problem related to RVM's installation of Ruby 2.3.3: compilation is failing, with details in a log file on the "websites2" machine, to which I don't have access. I'll contact Infra about it.
> > > >
> > > > In the meantime, I pinned the 8.x build to "websites1" and triggered a build, which succeeded. I left in the key download commands, so leaving them in permanently apparently wouldn't cause any trouble. (I'll make an issue to modify our build script to include a check for the keys and download them if they don't exist.)
> > > >
> > > > --
> > > > Steve
> > > >
> > > > > On Jun 27, 2019, at 9:22 AM, Cassandra Targett <ca...@gmail.com> wrote:
> > > > >
> > > > > Thanks Steve, that makes sense. Let me know if there’s anything I can do to help.
> > > > >
> > > > > If possible, maybe the script could check for the keys and import them if not available? At the very least, maybe I should add something about this scenario to internal Ref Guide docs. I totally forgot this had happened before.
> > > > >
> > > > > Sorry, I didn’t mean to imply you needed to disable the 8.1 job - I forgot to mention I recently gave myself permissions to do that in Jenkins - but thank you for doing it.
> > > > >
> > > > > Cassandra
> > > > > On Jun 27, 2019, 8:12 AM -0500, Steve Rowe <sa...@gmail.com>, wrote:
> > > > > > This has happened previously. Usually it's a new machine that has never built the ref guide previously. IIRC there was an Infra notice a couple days ago about one of the 2 machines used to build the ref guide ("website1" OSLT) being taken offline. Probably they put in place a replacement for it.
> > > > > >
> > > > > > In the past I've modified the job config to do what's suggested in the error message: download the key to the new machine, trigger a run, then comment out the key download in the job config.
> > > > > >
> > > > > > I suppose we could always download the key on every run?
> > > > > >
> > > > > > I'll work on it later today.
> > > > > >
> > > > > > I'll go disable the 8.1 ref guide job now.
> > > > > >
> > > > > > --
> > > > > > Steve
> > > > > >
> > > > > > > On Jun 27, 2019, at 8:02 AM, Cassandra Targett <ca...@gmail.com> wrote:
> > > > > > >
> > > > > > > This seems to be a persistent problem. It’s complaining about the keys for checking the RVM download. Right now it’s the 8.x job that’s failing (and 8.1, which needs to be disabled), but the master branch build appears to be fine.
> > > > > > >
> > > > > > > The jenkins.build.ref.guide.sh does the build, and I note in the beginning of the script that it seems to assume that the RVM package keys are already imported on the server (see https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/jenkins.build.ref.guide.sh#L13).
> > > > > > >
> > > > > > > Steve, you wrote this script and set up the overall build processes, do you think the problem here might be that some changes have been made on the Jenkins servers that removed those keys? The work to set this up was done in https://issues.apache.org/jira/browse/SOLR-10568, but it doesn’t mention the keys or how they got imported onto those machines, so that part isn’t clear to me.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Cassandra
> > > > > > > On Jun 27, 2019, 4:52 AM -0500, Apache Jenkins Server <je...@builds.apache.org>, wrote:
> > > > > > > > Build: https://builds.apache.org/job/Solr-reference-guide-8.x/3944/
> > > > > > > >
> > > > > > > > Log:
> > > > > > > > Started by timer
> > > > > > > > [EnvInject] - Loading node environment variables.
> > > > > > > > Building remotely on websites2 (git-websites svn-websites) in workspace /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-8.x
> > > > > > > > No credentials specified
> > > > > > > > > git rev-parse --is-inside-work-tree # timeout=10
> > > > > > > > Fetching changes from the remote Git repository
> > > > > > > > > git config remote.origin.url https://gitbox.apache.org/repos/asf/lucene-solr.git # timeout=10
> > > > > > > > Cleaning workspace
> > > > > > > > > git rev-parse --verify HEAD # timeout=10
> > > > > > > > Resetting working tree
> > > > > > > > > git reset --hard # timeout=10
> > > > > > > > > git clean -fdx # timeout=10
> > > > > > > > Fetching upstream changes from https://gitbox.apache.org/repos/asf/lucene-solr.git
> > > > > > > > > git --version # timeout=10
> > > > > > > > > git fetch --tags --progress https://gitbox.apache.org/repos/asf/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
> > > > > > > > > git rev-parse refs/remotes/origin/branch_8x^{commit} # timeout=10
> > > > > > > > > git rev-parse refs/remotes/origin/origin/branch_8x^{commit} # timeout=10
> > > > > > > > Checking out Revision 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db (refs/remotes/origin/branch_8x)
> > > > > > > > > git config core.sparsecheckout # timeout=10
> > > > > > > > > git checkout -f 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db
> > > > > > > > Commit message: "LUCENE-8886: Fix TestMutablePointsReaderUtils tests"
> > > > > > > > > git rev-list --no-walk cf443ad9f756407fa8db5ad5bfd39c26367acbc9 # timeout=10
> > > > > > > > No emails were triggered.
> > > > > > > > [Solr-reference-guide-8.x] $ /bin/bash -xe /tmp/jenkins9997498791593161379.sh
> > > > > > > > + bash dev-tools/scripts/jenkins.build.ref.guide.sh
> > > > > > > > + set -e
> > > > > > > > + RVM_PATH=/home/jenkins/.rvm
> > > > > > > > + RUBY_VERSION=ruby-2.3.3
> > > > > > > > + GEMSET=solr-refguide-gemset
> > > > > > > > + curl -sSL https://get.rvm.io
> > > > > > > > + bash -s -- --ignore-dotfiles stable
> > > > > > > > Turning on ignore dotfiles mode.
> > > > > > > > Downloading https://github.com/rvm/rvm/archive/1.29.8.tar.gz
> > > > > > > > Downloading https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc
> > > > > > > > gpg: Signature made Wed 08 May 2019 02:14:49 PM UTC
> > > > > > > > gpg: using RSA key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> > > > > > > > gpg: Can't check signature: No public key
> > > > > > > > GPG signature verification failed for '/home/jenkins/.rvm/archives/rvm-1.29.8.tgz' - 'https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc'! Try to install GPG v2 and then fetch the public key:
> > > > > > > >
> > > > > > > > gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> > > > > > > >
> > > > > > > > or if it fails:
> > > > > > > >
> > > > > > > > command curl -sSL https://rvm.io/mpapis.asc | gpg --import -
> > > > > > > > command curl -sSL https://rvm.io/pkuczynski.asc | gpg --import -
> > > > > > > >
> > > > > > > > In case of further problems with validation please refer to https://rvm.io/rvm/security
> > > > > > > >
> > > > > > > > Build step 'Execute shell' marked build as failure
> > > > > > > > Archiving artifacts
> > > > > > > > Publishing Javadoc
> > > > > > > > Email was triggered for: Failure - Any
> > > > > > > > Sending email for trigger: Failure - Any
> > > > > > > >
> > > > > > > >
> > > > > > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@lucene.apache.org
> > > > > >
> > > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
Re: [JENKINS] Solr-reference-guide-8.x - Build # 3944 - Still
Failing
Posted by Cassandra Targett <ca...@gmail.com>.
And the master job finished successfully (I checked and made sure it ran on the new server, and it did), so now I’ve backported the script change to branch_8x and updated the 8.x Ref Guide job to use the tag “git-websites” again. It just also finished successfully.
Thanks again Steve for your earlier help - I’ve let Infra know we’re all set in INFRA-18666.
Cassandra
On Jul 1, 2019, 10:27 AM -0500, Cassandra Targett <ca...@gmail.com>, wrote:
> I pushed a change to master and updated the Ref Guide master build to use “git-websites” again. I’ll monitor the next time it runs (~30 minutes from now) to see what happens. If it works, I’ll backport the change to branch_8x.
>
> Cassandra
> On Jun 30, 2019, 3:13 PM -0500, Erick Erickson <er...@gmail.com>, wrote:
> > AFAIC, just push it. Even if you do find a way to test it, we’d still potentially have something _else_ different between your test bed and websites2…..
> >
> > > On Jun 30, 2019, at 12:46 PM, Cassandra Targett <ca...@gmail.com> wrote:
> > >
> > > Thanks for filing those issues, Steve.
> > >
> > > The master branch Ref Guide build started failing this weekend with the Ruby installation problem, so I pinned that build to “websites1” for now.
> > >
> > > I saw the comments in the INFRA issue about using a newer version of Ruby, but wasn’t really sure how to test installing Ruby 2.5.1 instead. Apparently Gavin tried it and it worked all right after he initialized with the ant ivy-bootstrap target. Should I just commit the version change to the script, or try to test it some more first (somehow)?
> > >
> > > Cassandra
> > > On Jun 27, 2019, 10:41 AM -0500, Steve Rowe <sa...@gmail.com>, wrote:
> > > > INFRA JIRA for "websites2" problem: https://issues.apache.org/jira/browse/INFRA-18666
> > > >
> > > > JIRA to add GPG key download to the ref guide build script: https://issues.apache.org/jira/browse/SOLR-13582
> > > >
> > > > --
> > > > Steve
> > > >
> > > > > On Jun 27, 2019, at 10:50 AM, Steve Rowe <sa...@gmail.com> wrote:
> > > > >
> > > > > The 8.x ref guide builds are running on a node they've never run on before: "websites2". The master ref guide builds are running on the "websites1" node, which has not exhibited the same build problems. Both nodes are pinned to label "git-websites", which can result in builds on either the "websites1" node or the "websites2" node. AFAICT there's some kind of routing based on the identity (name maybe?) of the job, which results in effectively permanent assignment to either one or the other node.
> > > > >
> > > > > I put the key download commands from the error message into the 8.x job config and triggered a run. This enabled verification. But there is now another problem related to RVM's installation of Ruby 2.3.3: compilation is failing, with details in a log file on the "websites2" machine, to which I don't have access. I'll contact Infra about it.
> > > > >
> > > > > In the meantime, I pinned the 8.x build to "websites1" and triggered a build, which succeeded. I left in the key download commands, so leaving them in permanently apparently wouldn't cause any trouble. (I'll make an issue to modify our build script to include a check for the keys and download them if they don't exist.)
> > > > >
> > > > > --
> > > > > Steve
> > > > >
> > > > > > On Jun 27, 2019, at 9:22 AM, Cassandra Targett <ca...@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Steve, that makes sense. Let me know if there’s anything I can do to help.
> > > > > >
> > > > > > If possible, maybe the script could check for the keys and import them if not available? At the very least, maybe I should add something about this scenario to internal Ref Guide docs. I totally forgot this had happened before.
> > > > > >
> > > > > > Sorry, I didn’t mean to imply you needed to disable the 8.1 job - I forgot to mention I recently gave myself permissions to do that in Jenkins - but thank you for doing it.
> > > > > >
> > > > > > Cassandra
> > > > > > On Jun 27, 2019, 8:12 AM -0500, Steve Rowe <sa...@gmail.com>, wrote:
> > > > > > > This has happened previously. Usually it's a new machine that has never built the ref guide previously. IIRC there was an Infra notice a couple days ago about one of the 2 machines used to build the ref guide ("website1" OSLT) being taken offline. Probably they put in place a replacement for it.
> > > > > > >
> > > > > > > In the past I've modified the job config to do what's suggested in the error message: download the key to the new machine, trigger a run, then comment out the key download in the job config.
> > > > > > >
> > > > > > > I suppose we could always download the key on every run?
> > > > > > >
> > > > > > > I'll work on it later today.
> > > > > > >
> > > > > > > I'll go disable the 8.1 ref guide job now.
> > > > > > >
> > > > > > > --
> > > > > > > Steve
> > > > > > >
> > > > > > > > On Jun 27, 2019, at 8:02 AM, Cassandra Targett <ca...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > This seems to be a persistent problem. It’s complaining about the keys for checking the RVM download. Right now it’s the 8.x job that’s failing (and 8.1, which needs to be disabled), but the master branch build appears to be fine.
> > > > > > > >
> > > > > > > > The jenkins.build.ref.guide.sh does the build, and I note in the beginning of the script that it seems to assume that the RVM package keys are already imported on the server (see https://github.com/apache/lucene-solr/blob/master/dev-tools/scripts/jenkins.build.ref.guide.sh#L13).
> > > > > > > >
> > > > > > > > Steve, you wrote this script and set up the overall build processes, do you think the problem here might be that some changes have been made on the Jenkins servers that removed those keys? The work to set this up was done in https://issues.apache.org/jira/browse/SOLR-10568, but it doesn’t mention the keys or how they got imported onto those machines, so that part isn’t clear to me.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Cassandra
> > > > > > > > On Jun 27, 2019, 4:52 AM -0500, Apache Jenkins Server <je...@builds.apache.org>, wrote:
> > > > > > > > > Build: https://builds.apache.org/job/Solr-reference-guide-8.x/3944/
> > > > > > > > >
> > > > > > > > > Log:
> > > > > > > > > Started by timer
> > > > > > > > > [EnvInject] - Loading node environment variables.
> > > > > > > > > Building remotely on websites2 (git-websites svn-websites) in workspace /home/jenkins/jenkins-slave/workspace/Solr-reference-guide-8.x
> > > > > > > > > No credentials specified
> > > > > > > > > > git rev-parse --is-inside-work-tree # timeout=10
> > > > > > > > > Fetching changes from the remote Git repository
> > > > > > > > > > git config remote.origin.url https://gitbox.apache.org/repos/asf/lucene-solr.git # timeout=10
> > > > > > > > > Cleaning workspace
> > > > > > > > > > git rev-parse --verify HEAD # timeout=10
> > > > > > > > > Resetting working tree
> > > > > > > > > > git reset --hard # timeout=10
> > > > > > > > > > git clean -fdx # timeout=10
> > > > > > > > > Fetching upstream changes from https://gitbox.apache.org/repos/asf/lucene-solr.git
> > > > > > > > > > git --version # timeout=10
> > > > > > > > > > git fetch --tags --progress https://gitbox.apache.org/repos/asf/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
> > > > > > > > > > git rev-parse refs/remotes/origin/branch_8x^{commit} # timeout=10
> > > > > > > > > > git rev-parse refs/remotes/origin/origin/branch_8x^{commit} # timeout=10
> > > > > > > > > Checking out Revision 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db (refs/remotes/origin/branch_8x)
> > > > > > > > > > git config core.sparsecheckout # timeout=10
> > > > > > > > > > git checkout -f 4e58515b0c6aa297be2416c5e5d70e9eda4fb1db
> > > > > > > > > Commit message: "LUCENE-8886: Fix TestMutablePointsReaderUtils tests"
> > > > > > > > > > git rev-list --no-walk cf443ad9f756407fa8db5ad5bfd39c26367acbc9 # timeout=10
> > > > > > > > > No emails were triggered.
> > > > > > > > > [Solr-reference-guide-8.x] $ /bin/bash -xe /tmp/jenkins9997498791593161379.sh
> > > > > > > > > + bash dev-tools/scripts/jenkins.build.ref.guide.sh
> > > > > > > > > + set -e
> > > > > > > > > + RVM_PATH=/home/jenkins/.rvm
> > > > > > > > > + RUBY_VERSION=ruby-2.3.3
> > > > > > > > > + GEMSET=solr-refguide-gemset
> > > > > > > > > + curl -sSL https://get.rvm.io
> > > > > > > > > + bash -s -- --ignore-dotfiles stable
> > > > > > > > > Turning on ignore dotfiles mode.
> > > > > > > > > Downloading https://github.com/rvm/rvm/archive/1.29.8.tar.gz
> > > > > > > > > Downloading https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc
> > > > > > > > > gpg: Signature made Wed 08 May 2019 02:14:49 PM UTC
> > > > > > > > > gpg: using RSA key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> > > > > > > > > gpg: Can't check signature: No public key
> > > > > > > > > GPG signature verification failed for '/home/jenkins/.rvm/archives/rvm-1.29.8.tgz' - 'https://github.com/rvm/rvm/releases/download/1.29.8/1.29.8.tar.gz.asc'! Try to install GPG v2 and then fetch the public key:
> > > > > > > > >
> > > > > > > > > gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> > > > > > > > >
> > > > > > > > > or if it fails:
> > > > > > > > >
> > > > > > > > > command curl -sSL https://rvm.io/mpapis.asc | gpg --import -
> > > > > > > > > command curl -sSL https://rvm.io/pkuczynski.asc | gpg --import -
> > > > > > > > >
> > > > > > > > > In case of further problems with validation please refer to https://rvm.io/rvm/security
> > > > > > > > >
> > > > > > > > > Build step 'Execute shell' marked build as failure
> > > > > > > > > Archiving artifacts
> > > > > > > > > Publishing Javadoc
> > > > > > > > > Email was triggered for: Failure - Any
> > > > > > > > > Sending email for trigger: Failure - Any
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@lucene.apache.org
> > > > > > >
> > > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >