You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Jim King <Ji...@simplivity.com> on 2016/04/14 03:51:15 UTC
Build Failures
I'm still looking for answers on pull request build failures.
I have 2 or 3 PRs open right now and they've failed in the apache precommit builds for strange reasons.
The apache internal builds seem to be failing.
[Description: Description: simplivity-lg-xsmall]
James E. King, III
Architect
8 Technology Drive, 2nd Floor
Westborough, MA 01581-1756
Ph: 855-SVT-INFO
---------------------------------------------------------------------------------------
PRIVACY STATEMENT:
This message is a PRIVATE communication. This message and all attachments are a private communication sent by SimpliVity and are considered to be confidential or protected by privilege. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. Please notify the sender of the delivery error by replying to this message, and then delete it from your system.
---------------------------------------------------------------------------------------
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Wed, Apr 13, 2016 at 7:57 PM, John Sirois <js...@apache.org> wrote:
>
>
> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
>>
>>
>> On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
>> wrote:
>>
>>> I’m still looking for answers on pull request build failures.
>>>
>>> I have 2 or 3 PRs open right now and they’ve failed in the apache
>>> precommit builds for strange reasons.
>>>
>>> The apache internal builds seem to be failing.
>>>
>>
>> I think the answer is the breaks need a fixer; hopefully you can find
>> time to help fix.
>>
>> I say this because I started down a series of patches to the java
>> codegen/lib a while back and found a similar state - though on the pull
>> request builder (apache jenkins). I stopped my java stuff and fixed that
>> CI with the help of Aki and Jake reviewing and providing guidance. I am
>> not a thrift comitter.
>>
>
> I will say that its discouraging that that CI is now solid red too:
> https://builds.apache.org/job/Thrift-precommit/
> Part of the answer IMO is for committers to hold a hard line on accepting
> any patch, or pushing their own, w/o full green CIs.
>
FWIW, I just fixed the Thrift-precommit Jenkins 0th order problem, a bad
git config.
Although its now past git issues and building, I expect there will be more
failures since its been red so long - ie: presumably bad commits have made
it past while its been red.
>
>>
>>
>>>
>>>
>>> [image: Description: Description: simplivity-lg-xsmall]
>>>
>>> James E. King, III
>>>
>>> Architect
>>>
>>> 8 Technology Drive, 2nd Floor
>>> Westborough, MA 01581-1756
>>>
>>> Ph: 855-SVT-INFO
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> PRIVACY STATEMENT:
>>> This message is a PRIVATE communication. This message and all
>>> attachments are a private communication sent by SimpliVity and are
>>> considered to be confidential or protected by privilege. If you are not the
>>> intended recipient, you are hereby notified that any disclosure, copying,
>>> distribution or use of the information contained in or attached to this
>>> message is strictly prohibited. Please notify the sender of the delivery
>>> error by replying to this message, and then delete it from your system.
>>> For more information please visit http://www.simplivity.com
>>> ------------------------------
>>>
>>
>>
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Wed, Apr 13, 2016 at 8:34 PM, Jim King <Ji...@simplivity.com> wrote:
> The builds were failing claiming that a file was in the middle of being
> merged and they were all failing early.
> I think the build environment itself is compromised and there's nothing I
> can do about that.
>
Aha - my apologies. I read too quickly and/or merged other threads and
thought you were complaining about the Travis CI builds - which anyone can
try fixing.
Yeah - although not a comitter on thrift, I am one on aurora and have
Jenkins job admin rights - so I just fixed that git merge issue.
>
> -----Original Message-----
> From: John Sirois [mailto:jsirois@apache.org]
> Sent: Wednesday, April 13, 2016 9:58 PM
> To: dev@thrift.apache.org
> Subject: Re: Build Failures
>
> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
> >
> >
> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
> wrote:
> >
> >> I’m still looking for answers on pull request build failures.
> >>
> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
> >> precommit builds for strange reasons.
> >>
> >> The apache internal builds seem to be failing.
> >>
> >
> > I think the answer is the breaks need a fixer; hopefully you can find
> > time to help fix.
> >
> > I say this because I started down a series of patches to the java
> > codegen/lib a while back and found a similar state - though on the
> > pull request builder (apache jenkins). I stopped my java stuff and
> > fixed that CI with the help of Aki and Jake reviewing and providing
> > guidance. I am not a thrift comitter.
> >
>
> I will say that its discouraging that that CI is now solid red too:
> https://builds.apache.org/job/Thrift-precommit/
> Part of the answer IMO is for committers to hold a hard line on accepting
> any patch, or pushing their own, w/o full green CIs.
>
>
> >
> >
> >>
> >>
> >> [image: Description: Description: simplivity-lg-xsmall]
> >>
> >> James E. King, III
> >>
> >> Architect
> >>
> >> 8 Technology Drive, 2nd Floor
> >> Westborough, MA 01581-1756
> >>
> >> Ph: 855-SVT-INFO
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> PRIVACY STATEMENT:
> >> This message is a PRIVATE communication. This message and all
> >> attachments are a private communication sent by SimpliVity and are
> >> considered to be confidential or protected by privilege. If you are
> >> not the intended recipient, you are hereby notified that any
> >> disclosure, copying, distribution or use of the information contained
> >> in or attached to this message is strictly prohibited. Please notify
> >> the sender of the delivery error by replying to this message, and then
> delete it from your system.
> >> For more information please visit http://www.simplivity.com
> >> ------------------------------
> >>
> >
> >
>
> ---------------------------------------------------------------------------------------
> PRIVACY STATEMENT:
> This message is a PRIVATE communication. This message and all attachments
> are a private communication sent by SimpliVity and are considered to be
> confidential or protected by privilege. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of the information contained in or attached to this
> message is strictly prohibited. Please notify the sender of the delivery
> error by replying to this message, and then delete it from your system.
>
> ---------------------------------------------------------------------------------------
>
RE: Build Failures
Posted by Jim King <Ji...@simplivity.com>.
The builds were failing claiming that a file was in the middle of being merged and they were all failing early.
I think the build environment itself is compromised and there's nothing I can do about that.
-----Original Message-----
From: John Sirois [mailto:jsirois@apache.org]
Sent: Wednesday, April 13, 2016 9:58 PM
To: dev@thrift.apache.org
Subject: Re: Build Failures
On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
>
> On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com> wrote:
>
>> I’m still looking for answers on pull request build failures.
>>
>> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> precommit builds for strange reasons.
>>
>> The apache internal builds seem to be failing.
>>
>
> I think the answer is the breaks need a fixer; hopefully you can find
> time to help fix.
>
> I say this because I started down a series of patches to the java
> codegen/lib a while back and found a similar state - though on the
> pull request builder (apache jenkins). I stopped my java stuff and
> fixed that CI with the help of Aki and Jake reviewing and providing
> guidance. I am not a thrift comitter.
>
I will say that its discouraging that that CI is now solid red too:
https://builds.apache.org/job/Thrift-precommit/
Part of the answer IMO is for committers to hold a hard line on accepting any patch, or pushing their own, w/o full green CIs.
>
>
>>
>>
>> [image: Description: Description: simplivity-lg-xsmall]
>>
>> James E. King, III
>>
>> Architect
>>
>> 8 Technology Drive, 2nd Floor
>> Westborough, MA 01581-1756
>>
>> Ph: 855-SVT-INFO
>>
>>
>>
>>
>> ------------------------------
>> PRIVACY STATEMENT:
>> This message is a PRIVATE communication. This message and all
>> attachments are a private communication sent by SimpliVity and are
>> considered to be confidential or protected by privilege. If you are
>> not the intended recipient, you are hereby notified that any
>> disclosure, copying, distribution or use of the information contained
>> in or attached to this message is strictly prohibited. Please notify
>> the sender of the delivery error by replying to this message, and then delete it from your system.
>> For more information please visit http://www.simplivity.com
>> ------------------------------
>>
>
>
---------------------------------------------------------------------------------------
PRIVACY STATEMENT:
This message is a PRIVATE communication. This message and all attachments are a private communication sent by SimpliVity and are considered to be confidential or protected by privilege. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. Please notify the sender of the delivery error by replying to this message, and then delete it from your system.
---------------------------------------------------------------------------------------
Re: Build Failures
Posted by Aki Sukegawa <ns...@apache.org>.
Errors during docker download were actually Travis-CI problem.
They just fixed it.
https://github.com/travis-ci/travis-ci/issues/4924#issuecomment-213098154
We're still hitting the 50 min limit for cross test jobs often.
The other jobs seems stable now.
On Sun, Apr 17, 2016 at 6:58 AM Jens Geyer <je...@hotmail.com> wrote:
>
> Speaking about failing, I run into boost troubles again on my Suse machine
> with "make test" at cpp. It's the "lib-some-arbitrary-name.a not found"
> again. We really should do sth. about this.
>
> $0,02
> JensG
>
> -----Ursprüngliche Nachricht-----
> From: Jim King
> Sent: Saturday, April 16, 2016 3:40 PM
> To: dev@thrift.apache.org ; jsirois@apache.org
> Subject: RE: Build Failures
>
> Now it's failing a different way:
>
> https://builds.apache.org/job/Thrift-precommit/425/
>
> Merging refs/tags/changes/425
> > git rev-parse refs/tags/changes/425^{commit} # timeout=10
> FATAL: Command "git rev-parse refs/tags/changes/425^{commit}" returned
> status code 128:
> stdout: refs/tags/changes/425^{commit}
>
> stderr: fatal: ambiguous argument 'refs/tags/changes/425^{commit}': unknown
> revision or path not in the working tree.
> Use '--' to separate paths from revisions, like this:
> 'git <command> [<revision>...] -- [<file>...]'
>
> hudson.plugins.git.GitException: Command "git rev-parse
> refs/tags/changes/425^{commit}" returned status code 128:
> stdout: refs/tags/changes/425^{commit}
>
> stderr: fatal: ambiguous argument 'refs/tags/changes/425^{commit}': unknown
> revision or path not in the working tree.
> Use '--' to separate paths from revisions, like this:
> 'git <command> [<revision>...] -- [<file>...]'
>
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1669)
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1665)
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1307)
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1319)
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.revParse(CliGitAPIImpl.java:677)
> at hudson.plugins.git.GitAPI.revParse(GitAPI.java:316)
> at sun.reflect.GeneratedMethodAccessor1936.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> -----Original Message-----
> From: Jim King
> Sent: Thursday, April 14, 2016 10:14 AM
> To: 'dev@thrift.apache.org' <de...@thrift.apache.org>; 'jsirois@apache.org'
> <js...@apache.org>
> Subject: RE: Build Failures
>
> I got one build through (which failed in "d" tests) and now it's stuck in
> the same state, see:
> https://builds.apache.org/job/Thrift-precommit/411/
>
> FATAL: Could not checkout master with start point origin/master
> hudson.plugins.git.GitException: Could not checkout master with start point
> origin/master
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
> at
>
> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
> at
>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
> at
>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
> at
>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> at hudson.remoting.Request$2.run(Request.java:326)
> at
>
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> at ......remote call to H10(Native Method)
> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
> at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
> at hudson.remoting.Channel.call(Channel.java:781)
> at
>
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
> at
>
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
> at
>
> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
> at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
> at hudson.scm.SCM.checkout(SCM.java:485)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
> at
>
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
> at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> at
>
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
> at hudson.model.Run.execute(Run.java:1738)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:98)
> at hudson.model.Executor.run(Executor.java:410)
> Caused by: hudson.plugins.git.GitException: Command "git checkout -b master
> origin/master" returned status code 1:
> stdout: lib/lua/TCompactProtocol.lua: needs merge
>
> stderr: error: you need to resolve your current index first
>
> It looks like the build environment is not forced clean at the beginning of
> each build.
>
> - Jim
>
> -----Original Message-----
> From: Jim King
> Sent: Wednesday, April 13, 2016 10:34 PM
> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
> Subject: RE: Build Failures
>
> The builds were failing claiming that a file was in the middle of being
> merged and they were all failing early.
> I think the build environment itself is compromised and there's nothing I
> can do about that.
>
> -----Original Message-----
> From: John Sirois [mailto:jsirois@apache.org]
> Sent: Wednesday, April 13, 2016 9:58 PM
> To: dev@thrift.apache.org
> Subject: Re: Build Failures
>
> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
> >
> >
> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
> wrote:
> >
> >> I’m still looking for answers on pull request build failures.
> >>
> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
> >> precommit builds for strange reasons.
> >>
> >> The apache internal builds seem to be failing.
> >>
> >
> > I think the answer is the breaks need a fixer; hopefully you can find
> > time to help fix.
> >
> > I say this because I started down a series of patches to the java
> > codegen/lib a while back and found a similar state - though on the
> > pull request builder (apache jenkins). I stopped my java stuff and
> > fixed that CI with the help of Aki and Jake reviewing and providing
> > guidance. I am not a thrift comitter.
> >
>
> I will say that its discouraging that that CI is now solid red too:
> https://builds.apache.org/job/Thrift-precommit/
> Part of the answer IMO is for committers to hold a hard line on accepting
> any patch, or pushing their own, w/o full green CIs.
>
>
> >
> >
> >>
> >>
> >> [image: Description: Description: simplivity-lg-xsmall]
> >>
> >> James E. King, III
> >>
> >> Architect
> >>
> >> 8 Technology Drive, 2nd Floor
> >> Westborough, MA 01581-1756
> >>
> >> Ph: 855-SVT-INFO
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> PRIVACY STATEMENT:
> >> This message is a PRIVATE communication. This message and all
> >> attachments are a private communication sent by SimpliVity and are
> >> considered to be confidential or protected by privilege. If you are
> >> not the intended recipient, you are hereby notified that any
> >> disclosure, copying, distribution or use of the information contained
> >> in or attached to this message is strictly prohibited. Please notify
> >> the sender of the delivery error by replying to this message, and then
> >> delete it from your system.
> >> For more information please visit http://www.simplivity.com
> >> ------------------------------
> >>
> >
> >
>
> ---------------------------------------------------------------------------------------
> PRIVACY STATEMENT:
> This message is a PRIVATE communication. This message and all attachments
> are a private communication sent by SimpliVity and are considered to be
> confidential or protected by privilege. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of the information contained in or attached to this
> message is strictly prohibited. Please notify the sender of the delivery
> error by replying to this message, and then delete it from your system.
>
> ---------------------------------------------------------------------------------------
>
>
Re: Build Failures
Posted by Jens Geyer <je...@hotmail.com>.
Speaking about failing, I run into boost troubles again on my Suse machine
with "make test" at cpp. It's the "lib-some-arbitrary-name.a not found"
again. We really should do sth. about this.
$0,02
JensG
-----Ursprüngliche Nachricht-----
From: Jim King
Sent: Saturday, April 16, 2016 3:40 PM
To: dev@thrift.apache.org ; jsirois@apache.org
Subject: RE: Build Failures
Now it's failing a different way:
https://builds.apache.org/job/Thrift-precommit/425/
Merging refs/tags/changes/425
> git rev-parse refs/tags/changes/425^{commit} # timeout=10
FATAL: Command "git rev-parse refs/tags/changes/425^{commit}" returned
status code 128:
stdout: refs/tags/changes/425^{commit}
stderr: fatal: ambiguous argument 'refs/tags/changes/425^{commit}': unknown
revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
hudson.plugins.git.GitException: Command "git rev-parse
refs/tags/changes/425^{commit}" returned status code 128:
stdout: refs/tags/changes/425^{commit}
stderr: fatal: ambiguous argument 'refs/tags/changes/425^{commit}': unknown
revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1669)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1665)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1307)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1319)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.revParse(CliGitAPIImpl.java:677)
at hudson.plugins.git.GitAPI.revParse(GitAPI.java:316)
at sun.reflect.GeneratedMethodAccessor1936.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
-----Original Message-----
From: Jim King
Sent: Thursday, April 14, 2016 10:14 AM
To: 'dev@thrift.apache.org' <de...@thrift.apache.org>; 'jsirois@apache.org'
<js...@apache.org>
Subject: RE: Build Failures
I got one build through (which failed in "d" tests) and now it's stuck in
the same state, see:
https://builds.apache.org/job/Thrift-precommit/411/
FATAL: Could not checkout master with start point origin/master
hudson.plugins.git.GitException: Could not checkout master with start point
origin/master
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
at
org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
at
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
at
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to H10(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
at
org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
at
com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git checkout -b master
origin/master" returned status code 1:
stdout: lib/lua/TCompactProtocol.lua: needs merge
stderr: error: you need to resolve your current index first
It looks like the build environment is not forced clean at the beginning of
each build.
- Jim
-----Original Message-----
From: Jim King
Sent: Wednesday, April 13, 2016 10:34 PM
To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
Subject: RE: Build Failures
The builds were failing claiming that a file was in the middle of being
merged and they were all failing early.
I think the build environment itself is compromised and there's nothing I
can do about that.
-----Original Message-----
From: John Sirois [mailto:jsirois@apache.org]
Sent: Wednesday, April 13, 2016 9:58 PM
To: dev@thrift.apache.org
Subject: Re: Build Failures
On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
>
> On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com> wrote:
>
>> I’m still looking for answers on pull request build failures.
>>
>> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> precommit builds for strange reasons.
>>
>> The apache internal builds seem to be failing.
>>
>
> I think the answer is the breaks need a fixer; hopefully you can find
> time to help fix.
>
> I say this because I started down a series of patches to the java
> codegen/lib a while back and found a similar state - though on the
> pull request builder (apache jenkins). I stopped my java stuff and
> fixed that CI with the help of Aki and Jake reviewing and providing
> guidance. I am not a thrift comitter.
>
I will say that its discouraging that that CI is now solid red too:
https://builds.apache.org/job/Thrift-precommit/
Part of the answer IMO is for committers to hold a hard line on accepting
any patch, or pushing their own, w/o full green CIs.
>
>
>>
>>
>> [image: Description: Description: simplivity-lg-xsmall]
>>
>> James E. King, III
>>
>> Architect
>>
>> 8 Technology Drive, 2nd Floor
>> Westborough, MA 01581-1756
>>
>> Ph: 855-SVT-INFO
>>
>>
>>
>>
>> ------------------------------
>> PRIVACY STATEMENT:
>> This message is a PRIVATE communication. This message and all
>> attachments are a private communication sent by SimpliVity and are
>> considered to be confidential or protected by privilege. If you are
>> not the intended recipient, you are hereby notified that any
>> disclosure, copying, distribution or use of the information contained
>> in or attached to this message is strictly prohibited. Please notify
>> the sender of the delivery error by replying to this message, and then
>> delete it from your system.
>> For more information please visit http://www.simplivity.com
>> ------------------------------
>>
>
>
---------------------------------------------------------------------------------------
PRIVACY STATEMENT:
This message is a PRIVATE communication. This message and all attachments
are a private communication sent by SimpliVity and are considered to be
confidential or protected by privilege. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution or use of the information contained in or attached to this
message is strictly prohibited. Please notify the sender of the delivery
error by replying to this message, and then delete it from your system.
---------------------------------------------------------------------------------------
RE: Build Failures
Posted by Jim King <Ji...@simplivity.com>.
Now it's failing a different way:
https://builds.apache.org/job/Thrift-precommit/425/
Merging refs/tags/changes/425
> git rev-parse refs/tags/changes/425^{commit} # timeout=10
FATAL: Command "git rev-parse refs/tags/changes/425^{commit}" returned status code 128:
stdout: refs/tags/changes/425^{commit}
stderr: fatal: ambiguous argument 'refs/tags/changes/425^{commit}': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
hudson.plugins.git.GitException: Command "git rev-parse refs/tags/changes/425^{commit}" returned status code 128:
stdout: refs/tags/changes/425^{commit}
stderr: fatal: ambiguous argument 'refs/tags/changes/425^{commit}': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1669)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1665)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1307)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1319)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.revParse(CliGitAPIImpl.java:677)
at hudson.plugins.git.GitAPI.revParse(GitAPI.java:316)
at sun.reflect.GeneratedMethodAccessor1936.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
-----Original Message-----
From: Jim King
Sent: Thursday, April 14, 2016 10:14 AM
To: 'dev@thrift.apache.org' <de...@thrift.apache.org>; 'jsirois@apache.org' <js...@apache.org>
Subject: RE: Build Failures
I got one build through (which failed in "d" tests) and now it's stuck in the same state, see:
https://builds.apache.org/job/Thrift-precommit/411/
FATAL: Could not checkout master with start point origin/master
hudson.plugins.git.GitException: Could not checkout master with start point origin/master
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to H10(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
at com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git checkout -b master origin/master" returned status code 1:
stdout: lib/lua/TCompactProtocol.lua: needs merge
stderr: error: you need to resolve your current index first
It looks like the build environment is not forced clean at the beginning of each build.
- Jim
-----Original Message-----
From: Jim King
Sent: Wednesday, April 13, 2016 10:34 PM
To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
Subject: RE: Build Failures
The builds were failing claiming that a file was in the middle of being merged and they were all failing early.
I think the build environment itself is compromised and there's nothing I can do about that.
-----Original Message-----
From: John Sirois [mailto:jsirois@apache.org]
Sent: Wednesday, April 13, 2016 9:58 PM
To: dev@thrift.apache.org
Subject: Re: Build Failures
On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
>
> On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com> wrote:
>
>> I’m still looking for answers on pull request build failures.
>>
>> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> precommit builds for strange reasons.
>>
>> The apache internal builds seem to be failing.
>>
>
> I think the answer is the breaks need a fixer; hopefully you can find
> time to help fix.
>
> I say this because I started down a series of patches to the java
> codegen/lib a while back and found a similar state - though on the
> pull request builder (apache jenkins). I stopped my java stuff and
> fixed that CI with the help of Aki and Jake reviewing and providing
> guidance. I am not a thrift comitter.
>
I will say that its discouraging that that CI is now solid red too:
https://builds.apache.org/job/Thrift-precommit/
Part of the answer IMO is for committers to hold a hard line on accepting any patch, or pushing their own, w/o full green CIs.
>
>
>>
>>
>> [image: Description: Description: simplivity-lg-xsmall]
>>
>> James E. King, III
>>
>> Architect
>>
>> 8 Technology Drive, 2nd Floor
>> Westborough, MA 01581-1756
>>
>> Ph: 855-SVT-INFO
>>
>>
>>
>>
>> ------------------------------
>> PRIVACY STATEMENT:
>> This message is a PRIVATE communication. This message and all
>> attachments are a private communication sent by SimpliVity and are
>> considered to be confidential or protected by privilege. If you are
>> not the intended recipient, you are hereby notified that any
>> disclosure, copying, distribution or use of the information contained
>> in or attached to this message is strictly prohibited. Please notify
>> the sender of the delivery error by replying to this message, and then delete it from your system.
>> For more information please visit http://www.simplivity.com
>> ------------------------------
>>
>
>
---------------------------------------------------------------------------------------
PRIVACY STATEMENT:
This message is a PRIVATE communication. This message and all attachments are a private communication sent by SimpliVity and are considered to be confidential or protected by privilege. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. Please notify the sender of the delivery error by replying to this message, and then delete it from your system.
---------------------------------------------------------------------------------------
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 11:01 AM, Aki Sukegawa <ns...@apache.org> wrote:
> The most prominent source of failures are docker and npm downloads.
> I have an idea to reduce docker download failures but none for npm ones so
> far.
>
> For Jenkins, one problem is that the same code that worked is broken there
> in a few different ways recently.
> (I think it's running build on docker, am I correct ?)
> Again, it's fine we have different set of failures because it's useful to
> detect problems on various platforms.
> But it shouldn't start failing without code change like this.
>
> Another problem is that most committers don't have access to Jenkins job
> configuration, so it's very hard to fix in such occasions.
>
> @John I don't have permission to view the diff.
> You may want to share it here or on the JIRA issue you created.
>
If you don't have perms to view you won't have perms to fix. I know Jake
has perms. I think he should give Jenkins job admin perms to you and other
thrift committers interested in fixing CI.
He did this for me as an Aurora committer so I could work on the Aurora CI
some time ago and that's the only reason I happen to have perms (appears
Jenkins perms are global).
>
> On Fri, Apr 15, 2016 at 12:55 AM John Sirois <js...@apache.org> wrote:
>
>> On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jf...@apache.org>
>> wrote:
>>
>>> +1 to reducing this as much as possible so there is less maintenance
>>> overhead and setup. Ideally i would love to see all this dockerized so we
>>> are using a common base across the board and then either travis or jenkins
>>> or locally can all run those containers with the same setup in docker and
>>> the same outlined test scripts (which should be within the build folder) so
>>> this is the same repeatable process where ever it is run from
>>>
>>
>> I said this on another thread, but I think your and other comitter
>> limited resources should be bent towards doing this now and not accepting
>> patches, making changes or doing RCs.
>>
>> The thrift project feels untenable to contribute to with all the red CI.
>> After seeing this, I'm also growing disinclined towards depending on the
>> tool in my projects.
>> Maybe the thrift committers don't share this sense of priority, but to me
>> a stable green CI is the foundation of any project, without it everything
>> else crumbles.
>>
>>
>>>
>>>
>>>
>>> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org>
>>> wrote:
>>>
>>>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>>>>
>>>> > Quoting from my previous mail.
>>>> >
>>>> > > Other than Travis, make check is hanging for almost every build of
>>>> > Jenkins.
>>>> > > The log is not that clear but I think it's D test.
>>>> > > AFAIK the test was running fine a few weeks ago and nobody touched
>>>> it
>>>> > since then.
>>>> > > It might be due to insufficient resource on Jenkins.
>>>> >
>>>> > I suspect default task limit introduced in a recent version of docker
>>>> is
>>>> > not lifted on ASF jenkins.
>>>> >
>>>> > I'm not sure if it's worth maintaining sub-set of builds on another CI
>>>> > that has relatively unstable basis that cannot even be touched by
>>>> > committers.
>>>> > Less resource is fine because it can detect failures on such platforms
>>>> > like last time John enabled it.
>>>> > But it's apparently changing.
>>>> >
>>>>
>>>> Aha - that would be an interesting cause to the D hangs.
>>>>
>>>> I'm not clear on what you meant by the rest, but I assume you're
>>>> addressing
>>>> the confusing fact that thrift maintains 2 sets of broken CI jobs
>>>> (fwict)
>>>> for pull requests, TravisCI and Apache Jenkins.
>>>>
>>>> It seems to me 4 steps are needed to provide baseline sanity for
>>>> contributing to the project:
>>>> 1. Halt accepting and changes immediately.
>>>> 2. Pick Travis or Jenkins, kill the other.
>>>> 3. Get the winner from 2 green.
>>>> 4. Resume accepting patches that are green in CI and only green in CI.
>>>>
>>>>
>>>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org>
>>>> wrote:
>>>> >
>>>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Jim.King@simplivity.com
>>>> >
>>>> >> wrote:
>>>> >> >
>>>> >> >> I got one build through (which failed in "d" tests) and now it's
>>>> stuck
>>>> >> in
>>>> >> >> the same state, see:
>>>> >> >> https://builds.apache.org/job/Thrift-precommit/411/
>>>> >> >>
>>>> >> >> FATAL: Could not checkout master with start point origin/master
>>>> >> >> hudson.plugins.git.GitException: Could not checkout master with
>>>> start
>>>> >> >> point origin/master
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>>> >> >> at
>>>> hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>>> >> >> at
>>>> hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>>> >> >> at hudson.remoting.Request$2.run(Request.java:326)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>>> >> >> at
>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> >> >> at java.lang.Thread.run(Thread.java:745)
>>>> >> >> at ......remote call to H10(Native Method)
>>>> >> >> at
>>>> >> >>
>>>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>>> >> >> at
>>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>>> >> >> at hudson.remoting.Channel.call(Channel.java:781)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>>> >> >> at
>>>> >> >>
>>>> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>>> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>>> >> >> at
>>>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>>> >> >> at
>>>> >> >>
>>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>>> >> >> at hudson.model.Run.execute(Run.java:1738)
>>>> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>>> >> >> at
>>>> >> >>
>>>> hudson.model.ResourceController.execute(ResourceController.java:98)
>>>> >> >> at hudson.model.Executor.run(Executor.java:410)
>>>> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout
>>>> -b
>>>> >> >> master origin/master" returned status code 1:
>>>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>>> >> >>
>>>> >> >> stderr: error: you need to resolve your current index first
>>>> >> >>
>>>> >> >> It looks like the build environment is not forced clean at the
>>>> >> beginning
>>>> >> >> of each build.
>>>> >> >>
>>>> >> >
>>>> >> > Ack - looking now.
>>>> >> >
>>>> >> > It is odd that the git portion of these builds went sideways since
>>>> the
>>>> >> > Jenkins Job Config History auditing plugin shows the last change
>>>> >> (before my
>>>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>>>> >> plugins
>>>> >> > were updated by infra causing the previously working job config to
>>>> not
>>>> >> work
>>>> >> > any longer.
>>>> >> >
>>>> >>
>>>> >> OK - that analysis was wrong, clearly there has been a change in the
>>>> build
>>>> >> itself that modifies source code and this causes the issue.
>>>> >> I've enabled
>>>> <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>>> >> with
>>>> >> the following description:
>>>> >>
>>>> >> Clean up the workspace before every checkout by deleting all
>>>> untracked
>>>> >> files and directories, including those which are specified in
>>>> .gitignore.
>>>> >> It also resets all *tracked* files to their versioned state. This
>>>> ensures
>>>> >>
>>>> >> that the workspace is in the same state as if you cloned and checked
>>>> out
>>>> >> in
>>>> >> a brand-new empty directory, and ensures that your build is not
>>>> affected
>>>> >> by
>>>> >> the files generated by the previous build.
>>>> >>
>>>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>>>> which
>>>> >> should do it. That should insulate CI from bad tests that modify
>>>> checked
>>>> >> in
>>>> >> repo state, but those tests shouldn't exist either.
>>>> >>
>>>> >> COMMITTERS:
>>>> >> I'd like to reiterate to any committers out there that red CI must
>>>> be a
>>>> >> hard bright line that is not crossed when accepting patches;
>>>> otherwise
>>>> >> well
>>>> >> be right back here after getting this thing green again. Here we is
>>>> you -
>>>> >> I won't be interested in helping out a third time if this relapses.
>>>> >>
>>>> >>
>>>> >> >
>>>> >> >> - Jim
>>>> >> >>
>>>> >> >> -----Original Message-----
>>>> >> >> From: Jim King
>>>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>>> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <
>>>> jsirois@apache.org>
>>>> >> >> Subject: RE: Build Failures
>>>> >> >>
>>>> >> >> The builds were failing claiming that a file was in the middle of
>>>> being
>>>> >> >> merged and they were all failing early.
>>>> >> >> I think the build environment itself is compromised and there's
>>>> >> nothing I
>>>> >> >> can do about that.
>>>> >> >>
>>>> >> >> -----Original Message-----
>>>> >> >> From: John Sirois [mailto:jsirois@apache.org]
>>>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>>> >> >> To: dev@thrift.apache.org
>>>> >> >> Subject: Re: Build Failures
>>>> >> >>
>>>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>>>> >> wrote:
>>>> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <
>>>> Jim.King@simplivity.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> >> I’m still looking for answers on pull request build failures.
>>>> >> >> >>
>>>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the
>>>> apache
>>>> >> >> >> precommit builds for strange reasons.
>>>> >> >> >>
>>>> >> >> >> The apache internal builds seem to be failing.
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> > I think the answer is the breaks need a fixer; hopefully you
>>>> can find
>>>> >> >> > time to help fix.
>>>> >> >> >
>>>> >> >> > I say this because I started down a series of patches to the
>>>> java
>>>> >> >> > codegen/lib a while back and found a similar state - though on
>>>> the
>>>> >> >> > pull request builder (apache jenkins). I stopped my java stuff
>>>> and
>>>> >> >> > fixed that CI with the help of Aki and Jake reviewing and
>>>> providing
>>>> >> >> > guidance. I am not a thrift comitter.
>>>> >> >> >
>>>> >> >>
>>>> >> >> I will say that its discouraging that that CI is now solid red
>>>> too:
>>>> >> >> https://builds.apache.org/job/Thrift-precommit/
>>>> >> >> Part of the answer IMO is for committers to hold a hard line on
>>>> >> accepting
>>>> >> >> any patch, or pushing their own, w/o full green CIs.
>>>> >> >>
>>>> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>>> >> >> >>
>>>> >> >> >> James E. King, III
>>>> >> >> >>
>>>> >> >> >> Architect
>>>> >> >> >>
>>>> >> >> >> 8 Technology Drive, 2nd Floor
>>>> >> >> >> Westborough, MA 01581-1756
>>>> >> >> >>
>>>> >> >> >> Ph: 855-SVT-INFO
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> ------------------------------
>>>> >> >> >> PRIVACY STATEMENT:
>>>> >> >> >> This message is a PRIVATE communication. This message and all
>>>> >> >> >> attachments are a private communication sent by SimpliVity and
>>>> are
>>>> >> >> >> considered to be confidential or protected by privilege. If
>>>> you are
>>>> >> >> >> not the intended recipient, you are hereby notified that any
>>>> >> >> >> disclosure, copying, distribution or use of the information
>>>> >> contained
>>>> >> >> >> in or attached to this message is strictly prohibited. Please
>>>> notify
>>>> >> >> >> the sender of the delivery error by replying to this message,
>>>> and
>>>> >> then
>>>> >> >> delete it from your system.
>>>> >> >> >> For more information please visit http://www.simplivity.com
>>>> >> >> >> ------------------------------
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >>
>>>> >> >>
>>>> >>
>>>> ---------------------------------------------------------------------------------------
>>>> >> >> PRIVACY STATEMENT:
>>>> >> >> This message is a PRIVATE communication. This message and all
>>>> >> >> attachments are a private communication sent by SimpliVity and are
>>>> >> >> considered to be confidential or protected by privilege. If you
>>>> are
>>>> >> not the
>>>> >> >> intended recipient, you are hereby notified that any disclosure,
>>>> >> copying,
>>>> >> >> distribution or use of the information contained in or attached
>>>> to this
>>>> >> >> message is strictly prohibited. Please notify the sender of the
>>>> >> delivery
>>>> >> >> error by replying to this message, and then delete it from your
>>>> system.
>>>> >> >>
>>>> >> >>
>>>> >>
>>>> ---------------------------------------------------------------------------------------
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>
>>>
Re: Build Failures
Posted by Aki Sukegawa <ns...@apache.org>.
The most prominent source of failures are docker and npm downloads.
I have an idea to reduce docker download failures but none for npm ones so
far.
For Jenkins, one problem is that the same code that worked is broken there
in a few different ways recently.
(I think it's running build on docker, am I correct ?)
Again, it's fine we have different set of failures because it's useful to
detect problems on various platforms.
But it shouldn't start failing without code change like this.
Another problem is that most committers don't have access to Jenkins job
configuration, so it's very hard to fix in such occasions.
@John I don't have permission to view the diff.
You may want to share it here or on the JIRA issue you created.
On Fri, Apr 15, 2016 at 12:55 AM John Sirois <js...@apache.org> wrote:
> On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jf...@apache.org> wrote:
>
>> +1 to reducing this as much as possible so there is less maintenance
>> overhead and setup. Ideally i would love to see all this dockerized so we
>> are using a common base across the board and then either travis or jenkins
>> or locally can all run those containers with the same setup in docker and
>> the same outlined test scripts (which should be within the build folder) so
>> this is the same repeatable process where ever it is run from
>>
>
> I said this on another thread, but I think your and other comitter limited
> resources should be bent towards doing this now and not accepting patches,
> making changes or doing RCs.
>
> The thrift project feels untenable to contribute to with all the red CI.
> After seeing this, I'm also growing disinclined towards depending on the
> tool in my projects.
> Maybe the thrift committers don't share this sense of priority, but to me
> a stable green CI is the foundation of any project, without it everything
> else crumbles.
>
>
>>
>>
>>
>> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org> wrote:
>>
>>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>>>
>>> > Quoting from my previous mail.
>>> >
>>> > > Other than Travis, make check is hanging for almost every build of
>>> > Jenkins.
>>> > > The log is not that clear but I think it's D test.
>>> > > AFAIK the test was running fine a few weeks ago and nobody touched it
>>> > since then.
>>> > > It might be due to insufficient resource on Jenkins.
>>> >
>>> > I suspect default task limit introduced in a recent version of docker
>>> is
>>> > not lifted on ASF jenkins.
>>> >
>>> > I'm not sure if it's worth maintaining sub-set of builds on another CI
>>> > that has relatively unstable basis that cannot even be touched by
>>> > committers.
>>> > Less resource is fine because it can detect failures on such platforms
>>> > like last time John enabled it.
>>> > But it's apparently changing.
>>> >
>>>
>>> Aha - that would be an interesting cause to the D hangs.
>>>
>>> I'm not clear on what you meant by the rest, but I assume you're
>>> addressing
>>> the confusing fact that thrift maintains 2 sets of broken CI jobs (fwict)
>>> for pull requests, TravisCI and Apache Jenkins.
>>>
>>> It seems to me 4 steps are needed to provide baseline sanity for
>>> contributing to the project:
>>> 1. Halt accepting and changes immediately.
>>> 2. Pick Travis or Jenkins, kill the other.
>>> 3. Get the winner from 2 green.
>>> 4. Resume accepting patches that are green in CI and only green in CI.
>>>
>>>
>>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org>
>>> wrote:
>>> >
>>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>>> wrote:
>>> >>
>>> >> >
>>> >> >
>>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
>>> >> wrote:
>>> >> >
>>> >> >> I got one build through (which failed in "d" tests) and now it's
>>> stuck
>>> >> in
>>> >> >> the same state, see:
>>> >> >> https://builds.apache.org/job/Thrift-precommit/411/
>>> >> >>
>>> >> >> FATAL: Could not checkout master with start point origin/master
>>> >> >> hudson.plugins.git.GitException: Could not checkout master with
>>> start
>>> >> >> point origin/master
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> >> >> at
>>> >> >>
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> >> >> at
>>> >> >>
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>> >> >> at
>>> hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>> >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>> >> >> at hudson.remoting.Request$2.run(Request.java:326)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>> >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> >> >> at
>>> >> >>
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >> >> at
>>> >> >>
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >> >> at java.lang.Thread.run(Thread.java:745)
>>> >> >> at ......remote call to H10(Native Method)
>>> >> >> at
>>> >> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>> >> >> at
>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>> >> >> at hudson.remoting.Channel.call(Channel.java:781)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>> >> >> at
>>> >> >>
>>> >>
>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>> >> >> at
>>> >> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>> >> >> at
>>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>> >> >> at
>>> >> >>
>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>> >> >> at hudson.model.Run.execute(Run.java:1738)
>>> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>> >> >> at
>>> >> >> hudson.model.ResourceController.execute(ResourceController.java:98)
>>> >> >> at hudson.model.Executor.run(Executor.java:410)
>>> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout
>>> -b
>>> >> >> master origin/master" returned status code 1:
>>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>> >> >>
>>> >> >> stderr: error: you need to resolve your current index first
>>> >> >>
>>> >> >> It looks like the build environment is not forced clean at the
>>> >> beginning
>>> >> >> of each build.
>>> >> >>
>>> >> >
>>> >> > Ack - looking now.
>>> >> >
>>> >> > It is odd that the git portion of these builds went sideways since
>>> the
>>> >> > Jenkins Job Config History auditing plugin shows the last change
>>> >> (before my
>>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>>> >> plugins
>>> >> > were updated by infra causing the previously working job config to
>>> not
>>> >> work
>>> >> > any longer.
>>> >> >
>>> >>
>>> >> OK - that analysis was wrong, clearly there has been a change in the
>>> build
>>> >> itself that modifies source code and this causes the issue.
>>> >> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>> >> with
>>> >> the following description:
>>> >>
>>> >> Clean up the workspace before every checkout by deleting all untracked
>>> >> files and directories, including those which are specified in
>>> .gitignore.
>>> >> It also resets all *tracked* files to their versioned state. This
>>> ensures
>>> >>
>>> >> that the workspace is in the same state as if you cloned and checked
>>> out
>>> >> in
>>> >> a brand-new empty directory, and ensures that your build is not
>>> affected
>>> >> by
>>> >> the files generated by the previous build.
>>> >>
>>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>>> which
>>> >> should do it. That should insulate CI from bad tests that modify
>>> checked
>>> >> in
>>> >> repo state, but those tests shouldn't exist either.
>>> >>
>>> >> COMMITTERS:
>>> >> I'd like to reiterate to any committers out there that red CI must be
>>> a
>>> >> hard bright line that is not crossed when accepting patches; otherwise
>>> >> well
>>> >> be right back here after getting this thing green again. Here we is
>>> you -
>>> >> I won't be interested in helping out a third time if this relapses.
>>> >>
>>> >>
>>> >> >
>>> >> >> - Jim
>>> >> >>
>>> >> >> -----Original Message-----
>>> >> >> From: Jim King
>>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <
>>> jsirois@apache.org>
>>> >> >> Subject: RE: Build Failures
>>> >> >>
>>> >> >> The builds were failing claiming that a file was in the middle of
>>> being
>>> >> >> merged and they were all failing early.
>>> >> >> I think the build environment itself is compromised and there's
>>> >> nothing I
>>> >> >> can do about that.
>>> >> >>
>>> >> >> -----Original Message-----
>>> >> >> From: John Sirois [mailto:jsirois@apache.org]
>>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>> >> >> To: dev@thrift.apache.org
>>> >> >> Subject: Re: Build Failures
>>> >> >>
>>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>>> >> wrote:
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <
>>> Jim.King@simplivity.com>
>>> >> >> wrote:
>>> >> >> >
>>> >> >> >> I’m still looking for answers on pull request build failures.
>>> >> >> >>
>>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the
>>> apache
>>> >> >> >> precommit builds for strange reasons.
>>> >> >> >>
>>> >> >> >> The apache internal builds seem to be failing.
>>> >> >> >>
>>> >> >> >
>>> >> >> > I think the answer is the breaks need a fixer; hopefully you can
>>> find
>>> >> >> > time to help fix.
>>> >> >> >
>>> >> >> > I say this because I started down a series of patches to the java
>>> >> >> > codegen/lib a while back and found a similar state - though on
>>> the
>>> >> >> > pull request builder (apache jenkins). I stopped my java stuff
>>> and
>>> >> >> > fixed that CI with the help of Aki and Jake reviewing and
>>> providing
>>> >> >> > guidance. I am not a thrift comitter.
>>> >> >> >
>>> >> >>
>>> >> >> I will say that its discouraging that that CI is now solid red too:
>>> >> >> https://builds.apache.org/job/Thrift-precommit/
>>> >> >> Part of the answer IMO is for committers to hold a hard line on
>>> >> accepting
>>> >> >> any patch, or pushing their own, w/o full green CIs.
>>> >> >>
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>> >> >> >>
>>> >> >> >> James E. King, III
>>> >> >> >>
>>> >> >> >> Architect
>>> >> >> >>
>>> >> >> >> 8 Technology Drive, 2nd Floor
>>> >> >> >> Westborough, MA 01581-1756
>>> >> >> >>
>>> >> >> >> Ph: 855-SVT-INFO
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> ------------------------------
>>> >> >> >> PRIVACY STATEMENT:
>>> >> >> >> This message is a PRIVATE communication. This message and all
>>> >> >> >> attachments are a private communication sent by SimpliVity and
>>> are
>>> >> >> >> considered to be confidential or protected by privilege. If you
>>> are
>>> >> >> >> not the intended recipient, you are hereby notified that any
>>> >> >> >> disclosure, copying, distribution or use of the information
>>> >> contained
>>> >> >> >> in or attached to this message is strictly prohibited. Please
>>> notify
>>> >> >> >> the sender of the delivery error by replying to this message,
>>> and
>>> >> then
>>> >> >> delete it from your system.
>>> >> >> >> For more information please visit http://www.simplivity.com
>>> >> >> >> ------------------------------
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >>
>>> ---------------------------------------------------------------------------------------
>>> >> >> PRIVACY STATEMENT:
>>> >> >> This message is a PRIVATE communication. This message and all
>>> >> >> attachments are a private communication sent by SimpliVity and are
>>> >> >> considered to be confidential or protected by privilege. If you are
>>> >> not the
>>> >> >> intended recipient, you are hereby notified that any disclosure,
>>> >> copying,
>>> >> >> distribution or use of the information contained in or attached to
>>> this
>>> >> >> message is strictly prohibited. Please notify the sender of the
>>> >> delivery
>>> >> >> error by replying to this message, and then delete it from your
>>> system.
>>> >> >>
>>> >> >>
>>> >>
>>> ---------------------------------------------------------------------------------------
>>> >> >>
>>> >> >
>>> >> >
>>> >>
>>> >
>>>
>>
>>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 10:40 AM, John Sirois <js...@apache.org> wrote:
>
>
> On Thu, Apr 14, 2016 at 10:36 AM, Jake Farrell <jf...@apache.org>
> wrote:
>
>> completely agree, without passing CI we have no gauge on what impact a
>> patch could make and no guarantee that it will not destabilize the project.
>> I am a +1 to holding the 0.10.0 release candidate until we can get CI back
>> to green and kept that way
>>
>
> I'm very glad to hear this!
>
> I think I can be hands-off on the Jenkins Thrift-precommit job now.
> https://builds.apache.org/job/Thrift-precommit/418 is running with
> changes that include the equivalent of `git clean -fdx && git reset --hard
> HEAD` to get around the merge issue as well as more shimming in the docker
> image build to add a jenkins user to the image to avoid perms issues.
>
This would be the diff to examine for the committer who will pick this work
up:
https://builds.apache.org/job/Thrift-precommit/jobConfigHistory/showDiffFiles?timestamp1=2016-02-16_02-09-39×tamp2=2016-04-14_16-15-43
NB: I only enable checking out into a subdir to work around root owned
poisoned files in the root of the workspace. I don't have the necessary
perms to login to the jenkins machine and fix this; thus the workaround.
It could be undone by someone with perms.
>
>
>> -Jake
>>
>> On Thu, Apr 14, 2016 at 11:55 AM, John Sirois <js...@apache.org> wrote:
>>
>>>
>>>
>>> On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jf...@apache.org>
>>> wrote:
>>>
>>>> +1 to reducing this as much as possible so there is less maintenance
>>>> overhead and setup. Ideally i would love to see all this dockerized so we
>>>> are using a common base across the board and then either travis or jenkins
>>>> or locally can all run those containers with the same setup in docker and
>>>> the same outlined test scripts (which should be within the build folder) so
>>>> this is the same repeatable process where ever it is run from
>>>>
>>>
>>> I said this on another thread, but I think your and other comitter
>>> limited resources should be bent towards doing this now and not accepting
>>> patches, making changes or doing RCs.
>>>
>>> The thrift project feels untenable to contribute to with all the red
>>> CI. After seeing this, I'm also growing disinclined towards depending on
>>> the tool in my projects.
>>> Maybe the thrift committers don't share this sense of priority, but to
>>> me a stable green CI is the foundation of any project, without it
>>> everything else crumbles.
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org>
>>>> wrote:
>>>>
>>>>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org>
>>>>> wrote:
>>>>>
>>>>> > Quoting from my previous mail.
>>>>> >
>>>>> > > Other than Travis, make check is hanging for almost every build of
>>>>> > Jenkins.
>>>>> > > The log is not that clear but I think it's D test.
>>>>> > > AFAIK the test was running fine a few weeks ago and nobody touched
>>>>> it
>>>>> > since then.
>>>>> > > It might be due to insufficient resource on Jenkins.
>>>>> >
>>>>> > I suspect default task limit introduced in a recent version of
>>>>> docker is
>>>>> > not lifted on ASF jenkins.
>>>>> >
>>>>> > I'm not sure if it's worth maintaining sub-set of builds on another
>>>>> CI
>>>>> > that has relatively unstable basis that cannot even be touched by
>>>>> > committers.
>>>>> > Less resource is fine because it can detect failures on such
>>>>> platforms
>>>>> > like last time John enabled it.
>>>>> > But it's apparently changing.
>>>>> >
>>>>>
>>>>> Aha - that would be an interesting cause to the D hangs.
>>>>>
>>>>> I'm not clear on what you meant by the rest, but I assume you're
>>>>> addressing
>>>>> the confusing fact that thrift maintains 2 sets of broken CI jobs
>>>>> (fwict)
>>>>> for pull requests, TravisCI and Apache Jenkins.
>>>>>
>>>>> It seems to me 4 steps are needed to provide baseline sanity for
>>>>> contributing to the project:
>>>>> 1. Halt accepting and changes immediately.
>>>>> 2. Pick Travis or Jenkins, kill the other.
>>>>> 3. Get the winner from 2 green.
>>>>> 4. Resume accepting patches that are green in CI and only green in CI.
>>>>>
>>>>>
>>>>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org>
>>>>> wrote:
>>>>> >
>>>>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>>>>> wrote:
>>>>> >>
>>>>> >> >
>>>>> >> >
>>>>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <
>>>>> Jim.King@simplivity.com>
>>>>> >> wrote:
>>>>> >> >
>>>>> >> >> I got one build through (which failed in "d" tests) and now it's
>>>>> stuck
>>>>> >> in
>>>>> >> >> the same state, see:
>>>>> >> >> https://builds.apache.org/job/Thrift-precommit/411/
>>>>> >> >>
>>>>> >> >> FATAL: Could not checkout master with start point origin/master
>>>>> >> >> hudson.plugins.git.GitException: Could not checkout master with
>>>>> start
>>>>> >> >> point origin/master
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>>>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>>>> >> >> at
>>>>> hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>>>> >> >> at
>>>>> hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>>>> >> >> at hudson.remoting.Request$2.run(Request.java:326)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>>>> >> >> at
>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> >> >> at java.lang.Thread.run(Thread.java:745)
>>>>> >> >> at ......remote call to H10(Native Method)
>>>>> >> >> at
>>>>> >> >>
>>>>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>>>> >> >> at
>>>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>>>> >> >> at hudson.remoting.Channel.call(Channel.java:781)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>>>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>>>> >> >> at
>>>>> >> >>
>>>>> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>>>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>>>> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>>>> >> >> at
>>>>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>>>> >> >> at
>>>>> >> >>
>>>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>>>> >> >> at
>>>>> >> >>
>>>>> >>
>>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>>>> >> >> at hudson.model.Run.execute(Run.java:1738)
>>>>> >> >> at
>>>>> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>>>> >> >> at
>>>>> >> >>
>>>>> hudson.model.ResourceController.execute(ResourceController.java:98)
>>>>> >> >> at hudson.model.Executor.run(Executor.java:410)
>>>>> >> >> Caused by: hudson.plugins.git.GitException: Command "git
>>>>> checkout -b
>>>>> >> >> master origin/master" returned status code 1:
>>>>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>>>> >> >>
>>>>> >> >> stderr: error: you need to resolve your current index first
>>>>> >> >>
>>>>> >> >> It looks like the build environment is not forced clean at the
>>>>> >> beginning
>>>>> >> >> of each build.
>>>>> >> >>
>>>>> >> >
>>>>> >> > Ack - looking now.
>>>>> >> >
>>>>> >> > It is odd that the git portion of these builds went sideways
>>>>> since the
>>>>> >> > Jenkins Job Config History auditing plugin shows the last change
>>>>> >> (before my
>>>>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or
>>>>> its
>>>>> >> plugins
>>>>> >> > were updated by infra causing the previously working job config
>>>>> to not
>>>>> >> work
>>>>> >> > any longer.
>>>>> >> >
>>>>> >>
>>>>> >> OK - that analysis was wrong, clearly there has been a change in
>>>>> the build
>>>>> >> itself that modifies source code and this causes the issue.
>>>>> >> I've enabled
>>>>> <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>>>> >> with
>>>>> >> the following description:
>>>>> >>
>>>>> >> Clean up the workspace before every checkout by deleting all
>>>>> untracked
>>>>> >> files and directories, including those which are specified in
>>>>> .gitignore.
>>>>> >> It also resets all *tracked* files to their versioned state. This
>>>>> ensures
>>>>> >>
>>>>> >> that the workspace is in the same state as if you cloned and
>>>>> checked out
>>>>> >> in
>>>>> >> a brand-new empty directory, and ensures that your build is not
>>>>> affected
>>>>> >> by
>>>>> >> the files generated by the previous build.
>>>>> >>
>>>>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>>>>> which
>>>>> >> should do it. That should insulate CI from bad tests that modify
>>>>> checked
>>>>> >> in
>>>>> >> repo state, but those tests shouldn't exist either.
>>>>> >>
>>>>> >> COMMITTERS:
>>>>> >> I'd like to reiterate to any committers out there that red CI must
>>>>> be a
>>>>> >> hard bright line that is not crossed when accepting patches;
>>>>> otherwise
>>>>> >> well
>>>>> >> be right back here after getting this thing green again. Here we
>>>>> is you -
>>>>> >> I won't be interested in helping out a third time if this relapses.
>>>>> >>
>>>>> >>
>>>>> >> >
>>>>> >> >> - Jim
>>>>> >> >>
>>>>> >> >> -----Original Message-----
>>>>> >> >> From: Jim King
>>>>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>>>> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <
>>>>> jsirois@apache.org>
>>>>> >> >> Subject: RE: Build Failures
>>>>> >> >>
>>>>> >> >> The builds were failing claiming that a file was in the middle
>>>>> of being
>>>>> >> >> merged and they were all failing early.
>>>>> >> >> I think the build environment itself is compromised and there's
>>>>> >> nothing I
>>>>> >> >> can do about that.
>>>>> >> >>
>>>>> >> >> -----Original Message-----
>>>>> >> >> From: John Sirois [mailto:jsirois@apache.org]
>>>>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>>>> >> >> To: dev@thrift.apache.org
>>>>> >> >> Subject: Re: Build Failures
>>>>> >> >>
>>>>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <jsirois@apache.org
>>>>> >
>>>>> >> wrote:
>>>>> >> >>
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <
>>>>> Jim.King@simplivity.com>
>>>>> >> >> wrote:
>>>>> >> >> >
>>>>> >> >> >> I’m still looking for answers on pull request build failures.
>>>>> >> >> >>
>>>>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the
>>>>> apache
>>>>> >> >> >> precommit builds for strange reasons.
>>>>> >> >> >>
>>>>> >> >> >> The apache internal builds seem to be failing.
>>>>> >> >> >>
>>>>> >> >> >
>>>>> >> >> > I think the answer is the breaks need a fixer; hopefully you
>>>>> can find
>>>>> >> >> > time to help fix.
>>>>> >> >> >
>>>>> >> >> > I say this because I started down a series of patches to the
>>>>> java
>>>>> >> >> > codegen/lib a while back and found a similar state - though on
>>>>> the
>>>>> >> >> > pull request builder (apache jenkins). I stopped my java
>>>>> stuff and
>>>>> >> >> > fixed that CI with the help of Aki and Jake reviewing and
>>>>> providing
>>>>> >> >> > guidance. I am not a thrift comitter.
>>>>> >> >> >
>>>>> >> >>
>>>>> >> >> I will say that its discouraging that that CI is now solid red
>>>>> too:
>>>>> >> >> https://builds.apache.org/job/Thrift-precommit/
>>>>> >> >> Part of the answer IMO is for committers to hold a hard line on
>>>>> >> accepting
>>>>> >> >> any patch, or pushing their own, w/o full green CIs.
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> >>
>>>>> >> >> >>
>>>>> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>>>> >> >> >>
>>>>> >> >> >> James E. King, III
>>>>> >> >> >>
>>>>> >> >> >> Architect
>>>>> >> >> >>
>>>>> >> >> >> 8 Technology Drive, 2nd Floor
>>>>> >> >> >> Westborough, MA 01581-1756
>>>>> >> >> >>
>>>>> >> >> >> Ph: 855-SVT-INFO
>>>>> >> >> >>
>>>>> >> >> >>
>>>>> >> >> >>
>>>>> >> >> >>
>>>>> >> >> >> ------------------------------
>>>>> >> >> >> PRIVACY STATEMENT:
>>>>> >> >> >> This message is a PRIVATE communication. This message and all
>>>>> >> >> >> attachments are a private communication sent by SimpliVity
>>>>> and are
>>>>> >> >> >> considered to be confidential or protected by privilege. If
>>>>> you are
>>>>> >> >> >> not the intended recipient, you are hereby notified that any
>>>>> >> >> >> disclosure, copying, distribution or use of the information
>>>>> >> contained
>>>>> >> >> >> in or attached to this message is strictly prohibited. Please
>>>>> notify
>>>>> >> >> >> the sender of the delivery error by replying to this message,
>>>>> and
>>>>> >> then
>>>>> >> >> delete it from your system.
>>>>> >> >> >> For more information please visit http://www.simplivity.com
>>>>> >> >> >> ------------------------------
>>>>> >> >> >>
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >>
>>>>> >> >>
>>>>> >>
>>>>> ---------------------------------------------------------------------------------------
>>>>> >> >> PRIVACY STATEMENT:
>>>>> >> >> This message is a PRIVATE communication. This message and all
>>>>> >> >> attachments are a private communication sent by SimpliVity and
>>>>> are
>>>>> >> >> considered to be confidential or protected by privilege. If you
>>>>> are
>>>>> >> not the
>>>>> >> >> intended recipient, you are hereby notified that any disclosure,
>>>>> >> copying,
>>>>> >> >> distribution or use of the information contained in or attached
>>>>> to this
>>>>> >> >> message is strictly prohibited. Please notify the sender of the
>>>>> >> delivery
>>>>> >> >> error by replying to this message, and then delete it from your
>>>>> system.
>>>>> >> >>
>>>>> >> >>
>>>>> >>
>>>>> ---------------------------------------------------------------------------------------
>>>>> >> >>
>>>>> >> >
>>>>> >> >
>>>>> >>
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 10:36 AM, Jake Farrell <jf...@apache.org> wrote:
> completely agree, without passing CI we have no gauge on what impact a
> patch could make and no guarantee that it will not destabilize the project.
> I am a +1 to holding the 0.10.0 release candidate until we can get CI back
> to green and kept that way
>
I'm very glad to hear this!
I think I can be hands-off on the Jenkins Thrift-precommit job now.
https://builds.apache.org/job/Thrift-precommit/418 is running with changes
that include the equivalent of `git clean -fdx && git reset --hard HEAD` to
get around the merge issue as well as more shimming in the docker image
build to add a jenkins user to the image to avoid perms issues.
> -Jake
>
> On Thu, Apr 14, 2016 at 11:55 AM, John Sirois <js...@apache.org> wrote:
>
>>
>>
>> On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jf...@apache.org>
>> wrote:
>>
>>> +1 to reducing this as much as possible so there is less maintenance
>>> overhead and setup. Ideally i would love to see all this dockerized so we
>>> are using a common base across the board and then either travis or jenkins
>>> or locally can all run those containers with the same setup in docker and
>>> the same outlined test scripts (which should be within the build folder) so
>>> this is the same repeatable process where ever it is run from
>>>
>>
>> I said this on another thread, but I think your and other comitter
>> limited resources should be bent towards doing this now and not accepting
>> patches, making changes or doing RCs.
>>
>> The thrift project feels untenable to contribute to with all the red CI.
>> After seeing this, I'm also growing disinclined towards depending on the
>> tool in my projects.
>> Maybe the thrift committers don't share this sense of priority, but to me
>> a stable green CI is the foundation of any project, without it everything
>> else crumbles.
>>
>>
>>>
>>>
>>>
>>> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org>
>>> wrote:
>>>
>>>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>>>>
>>>> > Quoting from my previous mail.
>>>> >
>>>> > > Other than Travis, make check is hanging for almost every build of
>>>> > Jenkins.
>>>> > > The log is not that clear but I think it's D test.
>>>> > > AFAIK the test was running fine a few weeks ago and nobody touched
>>>> it
>>>> > since then.
>>>> > > It might be due to insufficient resource on Jenkins.
>>>> >
>>>> > I suspect default task limit introduced in a recent version of docker
>>>> is
>>>> > not lifted on ASF jenkins.
>>>> >
>>>> > I'm not sure if it's worth maintaining sub-set of builds on another CI
>>>> > that has relatively unstable basis that cannot even be touched by
>>>> > committers.
>>>> > Less resource is fine because it can detect failures on such platforms
>>>> > like last time John enabled it.
>>>> > But it's apparently changing.
>>>> >
>>>>
>>>> Aha - that would be an interesting cause to the D hangs.
>>>>
>>>> I'm not clear on what you meant by the rest, but I assume you're
>>>> addressing
>>>> the confusing fact that thrift maintains 2 sets of broken CI jobs
>>>> (fwict)
>>>> for pull requests, TravisCI and Apache Jenkins.
>>>>
>>>> It seems to me 4 steps are needed to provide baseline sanity for
>>>> contributing to the project:
>>>> 1. Halt accepting and changes immediately.
>>>> 2. Pick Travis or Jenkins, kill the other.
>>>> 3. Get the winner from 2 green.
>>>> 4. Resume accepting patches that are green in CI and only green in CI.
>>>>
>>>>
>>>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org>
>>>> wrote:
>>>> >
>>>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Jim.King@simplivity.com
>>>> >
>>>> >> wrote:
>>>> >> >
>>>> >> >> I got one build through (which failed in "d" tests) and now it's
>>>> stuck
>>>> >> in
>>>> >> >> the same state, see:
>>>> >> >> https://builds.apache.org/job/Thrift-precommit/411/
>>>> >> >>
>>>> >> >> FATAL: Could not checkout master with start point origin/master
>>>> >> >> hudson.plugins.git.GitException: Could not checkout master with
>>>> start
>>>> >> >> point origin/master
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>>> >> >> at
>>>> hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>>> >> >> at
>>>> hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>>> >> >> at hudson.remoting.Request$2.run(Request.java:326)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>>> >> >> at
>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> >> >> at java.lang.Thread.run(Thread.java:745)
>>>> >> >> at ......remote call to H10(Native Method)
>>>> >> >> at
>>>> >> >>
>>>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>>> >> >> at
>>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>>> >> >> at hudson.remoting.Channel.call(Channel.java:781)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>>> >> >> at
>>>> >> >>
>>>> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>>> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>>> >> >> at
>>>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>>> >> >> at
>>>> >> >>
>>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>>> >> >> at
>>>> >> >>
>>>> >>
>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>>> >> >> at hudson.model.Run.execute(Run.java:1738)
>>>> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>>> >> >> at
>>>> >> >>
>>>> hudson.model.ResourceController.execute(ResourceController.java:98)
>>>> >> >> at hudson.model.Executor.run(Executor.java:410)
>>>> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout
>>>> -b
>>>> >> >> master origin/master" returned status code 1:
>>>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>>> >> >>
>>>> >> >> stderr: error: you need to resolve your current index first
>>>> >> >>
>>>> >> >> It looks like the build environment is not forced clean at the
>>>> >> beginning
>>>> >> >> of each build.
>>>> >> >>
>>>> >> >
>>>> >> > Ack - looking now.
>>>> >> >
>>>> >> > It is odd that the git portion of these builds went sideways since
>>>> the
>>>> >> > Jenkins Job Config History auditing plugin shows the last change
>>>> >> (before my
>>>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>>>> >> plugins
>>>> >> > were updated by infra causing the previously working job config to
>>>> not
>>>> >> work
>>>> >> > any longer.
>>>> >> >
>>>> >>
>>>> >> OK - that analysis was wrong, clearly there has been a change in the
>>>> build
>>>> >> itself that modifies source code and this causes the issue.
>>>> >> I've enabled
>>>> <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>>> >> with
>>>> >> the following description:
>>>> >>
>>>> >> Clean up the workspace before every checkout by deleting all
>>>> untracked
>>>> >> files and directories, including those which are specified in
>>>> .gitignore.
>>>> >> It also resets all *tracked* files to their versioned state. This
>>>> ensures
>>>> >>
>>>> >> that the workspace is in the same state as if you cloned and checked
>>>> out
>>>> >> in
>>>> >> a brand-new empty directory, and ensures that your build is not
>>>> affected
>>>> >> by
>>>> >> the files generated by the previous build.
>>>> >>
>>>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>>>> which
>>>> >> should do it. That should insulate CI from bad tests that modify
>>>> checked
>>>> >> in
>>>> >> repo state, but those tests shouldn't exist either.
>>>> >>
>>>> >> COMMITTERS:
>>>> >> I'd like to reiterate to any committers out there that red CI must
>>>> be a
>>>> >> hard bright line that is not crossed when accepting patches;
>>>> otherwise
>>>> >> well
>>>> >> be right back here after getting this thing green again. Here we is
>>>> you -
>>>> >> I won't be interested in helping out a third time if this relapses.
>>>> >>
>>>> >>
>>>> >> >
>>>> >> >> - Jim
>>>> >> >>
>>>> >> >> -----Original Message-----
>>>> >> >> From: Jim King
>>>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>>> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <
>>>> jsirois@apache.org>
>>>> >> >> Subject: RE: Build Failures
>>>> >> >>
>>>> >> >> The builds were failing claiming that a file was in the middle of
>>>> being
>>>> >> >> merged and they were all failing early.
>>>> >> >> I think the build environment itself is compromised and there's
>>>> >> nothing I
>>>> >> >> can do about that.
>>>> >> >>
>>>> >> >> -----Original Message-----
>>>> >> >> From: John Sirois [mailto:jsirois@apache.org]
>>>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>>> >> >> To: dev@thrift.apache.org
>>>> >> >> Subject: Re: Build Failures
>>>> >> >>
>>>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>>>> >> wrote:
>>>> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <
>>>> Jim.King@simplivity.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> >> I’m still looking for answers on pull request build failures.
>>>> >> >> >>
>>>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the
>>>> apache
>>>> >> >> >> precommit builds for strange reasons.
>>>> >> >> >>
>>>> >> >> >> The apache internal builds seem to be failing.
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> > I think the answer is the breaks need a fixer; hopefully you
>>>> can find
>>>> >> >> > time to help fix.
>>>> >> >> >
>>>> >> >> > I say this because I started down a series of patches to the
>>>> java
>>>> >> >> > codegen/lib a while back and found a similar state - though on
>>>> the
>>>> >> >> > pull request builder (apache jenkins). I stopped my java stuff
>>>> and
>>>> >> >> > fixed that CI with the help of Aki and Jake reviewing and
>>>> providing
>>>> >> >> > guidance. I am not a thrift comitter.
>>>> >> >> >
>>>> >> >>
>>>> >> >> I will say that its discouraging that that CI is now solid red
>>>> too:
>>>> >> >> https://builds.apache.org/job/Thrift-precommit/
>>>> >> >> Part of the answer IMO is for committers to hold a hard line on
>>>> >> accepting
>>>> >> >> any patch, or pushing their own, w/o full green CIs.
>>>> >> >>
>>>> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>>> >> >> >>
>>>> >> >> >> James E. King, III
>>>> >> >> >>
>>>> >> >> >> Architect
>>>> >> >> >>
>>>> >> >> >> 8 Technology Drive, 2nd Floor
>>>> >> >> >> Westborough, MA 01581-1756
>>>> >> >> >>
>>>> >> >> >> Ph: 855-SVT-INFO
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> ------------------------------
>>>> >> >> >> PRIVACY STATEMENT:
>>>> >> >> >> This message is a PRIVATE communication. This message and all
>>>> >> >> >> attachments are a private communication sent by SimpliVity and
>>>> are
>>>> >> >> >> considered to be confidential or protected by privilege. If
>>>> you are
>>>> >> >> >> not the intended recipient, you are hereby notified that any
>>>> >> >> >> disclosure, copying, distribution or use of the information
>>>> >> contained
>>>> >> >> >> in or attached to this message is strictly prohibited. Please
>>>> notify
>>>> >> >> >> the sender of the delivery error by replying to this message,
>>>> and
>>>> >> then
>>>> >> >> delete it from your system.
>>>> >> >> >> For more information please visit http://www.simplivity.com
>>>> >> >> >> ------------------------------
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >>
>>>> >> >>
>>>> >>
>>>> ---------------------------------------------------------------------------------------
>>>> >> >> PRIVACY STATEMENT:
>>>> >> >> This message is a PRIVATE communication. This message and all
>>>> >> >> attachments are a private communication sent by SimpliVity and are
>>>> >> >> considered to be confidential or protected by privilege. If you
>>>> are
>>>> >> not the
>>>> >> >> intended recipient, you are hereby notified that any disclosure,
>>>> >> copying,
>>>> >> >> distribution or use of the information contained in or attached
>>>> to this
>>>> >> >> message is strictly prohibited. Please notify the sender of the
>>>> >> delivery
>>>> >> >> error by replying to this message, and then delete it from your
>>>> system.
>>>> >> >>
>>>> >> >>
>>>> >>
>>>> ---------------------------------------------------------------------------------------
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>
Re: Build Failures
Posted by Jake Farrell <jf...@apache.org>.
completely agree, without passing CI we have no gauge on what impact a
patch could make and no guarantee that it will not destabilize the project.
I am a +1 to holding the 0.10.0 release candidate until we can get CI back
to green and kept that way
-Jake
On Thu, Apr 14, 2016 at 11:55 AM, John Sirois <js...@apache.org> wrote:
>
>
> On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jf...@apache.org> wrote:
>
>> +1 to reducing this as much as possible so there is less maintenance
>> overhead and setup. Ideally i would love to see all this dockerized so we
>> are using a common base across the board and then either travis or jenkins
>> or locally can all run those containers with the same setup in docker and
>> the same outlined test scripts (which should be within the build folder) so
>> this is the same repeatable process where ever it is run from
>>
>
> I said this on another thread, but I think your and other comitter limited
> resources should be bent towards doing this now and not accepting patches,
> making changes or doing RCs.
>
> The thrift project feels untenable to contribute to with all the red CI.
> After seeing this, I'm also growing disinclined towards depending on the
> tool in my projects.
> Maybe the thrift committers don't share this sense of priority, but to me
> a stable green CI is the foundation of any project, without it everything
> else crumbles.
>
>
>>
>>
>>
>> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org> wrote:
>>
>>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>>>
>>> > Quoting from my previous mail.
>>> >
>>> > > Other than Travis, make check is hanging for almost every build of
>>> > Jenkins.
>>> > > The log is not that clear but I think it's D test.
>>> > > AFAIK the test was running fine a few weeks ago and nobody touched it
>>> > since then.
>>> > > It might be due to insufficient resource on Jenkins.
>>> >
>>> > I suspect default task limit introduced in a recent version of docker
>>> is
>>> > not lifted on ASF jenkins.
>>> >
>>> > I'm not sure if it's worth maintaining sub-set of builds on another CI
>>> > that has relatively unstable basis that cannot even be touched by
>>> > committers.
>>> > Less resource is fine because it can detect failures on such platforms
>>> > like last time John enabled it.
>>> > But it's apparently changing.
>>> >
>>>
>>> Aha - that would be an interesting cause to the D hangs.
>>>
>>> I'm not clear on what you meant by the rest, but I assume you're
>>> addressing
>>> the confusing fact that thrift maintains 2 sets of broken CI jobs (fwict)
>>> for pull requests, TravisCI and Apache Jenkins.
>>>
>>> It seems to me 4 steps are needed to provide baseline sanity for
>>> contributing to the project:
>>> 1. Halt accepting and changes immediately.
>>> 2. Pick Travis or Jenkins, kill the other.
>>> 3. Get the winner from 2 green.
>>> 4. Resume accepting patches that are green in CI and only green in CI.
>>>
>>>
>>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org>
>>> wrote:
>>> >
>>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>>> wrote:
>>> >>
>>> >> >
>>> >> >
>>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
>>> >> wrote:
>>> >> >
>>> >> >> I got one build through (which failed in "d" tests) and now it's
>>> stuck
>>> >> in
>>> >> >> the same state, see:
>>> >> >> https://builds.apache.org/job/Thrift-precommit/411/
>>> >> >>
>>> >> >> FATAL: Could not checkout master with start point origin/master
>>> >> >> hudson.plugins.git.GitException: Could not checkout master with
>>> start
>>> >> >> point origin/master
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> >> >> at
>>> >> >>
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> >> >> at
>>> >> >>
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>> >> >> at
>>> hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>> >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>> >> >> at hudson.remoting.Request$2.run(Request.java:326)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>> >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> >> >> at
>>> >> >>
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >> >> at
>>> >> >>
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >> >> at java.lang.Thread.run(Thread.java:745)
>>> >> >> at ......remote call to H10(Native Method)
>>> >> >> at
>>> >> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>> >> >> at
>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>> >> >> at hudson.remoting.Channel.call(Channel.java:781)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>> >> >> at
>>> >> >>
>>> >>
>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>> >> >> at
>>> >> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>> >> >> at
>>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>> >> >> at
>>> >> >>
>>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>> >> >> at
>>> >> >>
>>> >>
>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>> >> >> at hudson.model.Run.execute(Run.java:1738)
>>> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>> >> >> at
>>> >> >> hudson.model.ResourceController.execute(ResourceController.java:98)
>>> >> >> at hudson.model.Executor.run(Executor.java:410)
>>> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout
>>> -b
>>> >> >> master origin/master" returned status code 1:
>>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>> >> >>
>>> >> >> stderr: error: you need to resolve your current index first
>>> >> >>
>>> >> >> It looks like the build environment is not forced clean at the
>>> >> beginning
>>> >> >> of each build.
>>> >> >>
>>> >> >
>>> >> > Ack - looking now.
>>> >> >
>>> >> > It is odd that the git portion of these builds went sideways since
>>> the
>>> >> > Jenkins Job Config History auditing plugin shows the last change
>>> >> (before my
>>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>>> >> plugins
>>> >> > were updated by infra causing the previously working job config to
>>> not
>>> >> work
>>> >> > any longer.
>>> >> >
>>> >>
>>> >> OK - that analysis was wrong, clearly there has been a change in the
>>> build
>>> >> itself that modifies source code and this causes the issue.
>>> >> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>> >> with
>>> >> the following description:
>>> >>
>>> >> Clean up the workspace before every checkout by deleting all untracked
>>> >> files and directories, including those which are specified in
>>> .gitignore.
>>> >> It also resets all *tracked* files to their versioned state. This
>>> ensures
>>> >>
>>> >> that the workspace is in the same state as if you cloned and checked
>>> out
>>> >> in
>>> >> a brand-new empty directory, and ensures that your build is not
>>> affected
>>> >> by
>>> >> the files generated by the previous build.
>>> >>
>>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>>> which
>>> >> should do it. That should insulate CI from bad tests that modify
>>> checked
>>> >> in
>>> >> repo state, but those tests shouldn't exist either.
>>> >>
>>> >> COMMITTERS:
>>> >> I'd like to reiterate to any committers out there that red CI must be
>>> a
>>> >> hard bright line that is not crossed when accepting patches; otherwise
>>> >> well
>>> >> be right back here after getting this thing green again. Here we is
>>> you -
>>> >> I won't be interested in helping out a third time if this relapses.
>>> >>
>>> >>
>>> >> >
>>> >> >> - Jim
>>> >> >>
>>> >> >> -----Original Message-----
>>> >> >> From: Jim King
>>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <
>>> jsirois@apache.org>
>>> >> >> Subject: RE: Build Failures
>>> >> >>
>>> >> >> The builds were failing claiming that a file was in the middle of
>>> being
>>> >> >> merged and they were all failing early.
>>> >> >> I think the build environment itself is compromised and there's
>>> >> nothing I
>>> >> >> can do about that.
>>> >> >>
>>> >> >> -----Original Message-----
>>> >> >> From: John Sirois [mailto:jsirois@apache.org]
>>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>> >> >> To: dev@thrift.apache.org
>>> >> >> Subject: Re: Build Failures
>>> >> >>
>>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>>> >> wrote:
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <
>>> Jim.King@simplivity.com>
>>> >> >> wrote:
>>> >> >> >
>>> >> >> >> I’m still looking for answers on pull request build failures.
>>> >> >> >>
>>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the
>>> apache
>>> >> >> >> precommit builds for strange reasons.
>>> >> >> >>
>>> >> >> >> The apache internal builds seem to be failing.
>>> >> >> >>
>>> >> >> >
>>> >> >> > I think the answer is the breaks need a fixer; hopefully you can
>>> find
>>> >> >> > time to help fix.
>>> >> >> >
>>> >> >> > I say this because I started down a series of patches to the java
>>> >> >> > codegen/lib a while back and found a similar state - though on
>>> the
>>> >> >> > pull request builder (apache jenkins). I stopped my java stuff
>>> and
>>> >> >> > fixed that CI with the help of Aki and Jake reviewing and
>>> providing
>>> >> >> > guidance. I am not a thrift comitter.
>>> >> >> >
>>> >> >>
>>> >> >> I will say that its discouraging that that CI is now solid red too:
>>> >> >> https://builds.apache.org/job/Thrift-precommit/
>>> >> >> Part of the answer IMO is for committers to hold a hard line on
>>> >> accepting
>>> >> >> any patch, or pushing their own, w/o full green CIs.
>>> >> >>
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>> >> >> >>
>>> >> >> >> James E. King, III
>>> >> >> >>
>>> >> >> >> Architect
>>> >> >> >>
>>> >> >> >> 8 Technology Drive, 2nd Floor
>>> >> >> >> Westborough, MA 01581-1756
>>> >> >> >>
>>> >> >> >> Ph: 855-SVT-INFO
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> ------------------------------
>>> >> >> >> PRIVACY STATEMENT:
>>> >> >> >> This message is a PRIVATE communication. This message and all
>>> >> >> >> attachments are a private communication sent by SimpliVity and
>>> are
>>> >> >> >> considered to be confidential or protected by privilege. If you
>>> are
>>> >> >> >> not the intended recipient, you are hereby notified that any
>>> >> >> >> disclosure, copying, distribution or use of the information
>>> >> contained
>>> >> >> >> in or attached to this message is strictly prohibited. Please
>>> notify
>>> >> >> >> the sender of the delivery error by replying to this message,
>>> and
>>> >> then
>>> >> >> delete it from your system.
>>> >> >> >> For more information please visit http://www.simplivity.com
>>> >> >> >> ------------------------------
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >>
>>> ---------------------------------------------------------------------------------------
>>> >> >> PRIVACY STATEMENT:
>>> >> >> This message is a PRIVATE communication. This message and all
>>> >> >> attachments are a private communication sent by SimpliVity and are
>>> >> >> considered to be confidential or protected by privilege. If you are
>>> >> not the
>>> >> >> intended recipient, you are hereby notified that any disclosure,
>>> >> copying,
>>> >> >> distribution or use of the information contained in or attached to
>>> this
>>> >> >> message is strictly prohibited. Please notify the sender of the
>>> >> delivery
>>> >> >> error by replying to this message, and then delete it from your
>>> system.
>>> >> >>
>>> >> >>
>>> >>
>>> ---------------------------------------------------------------------------------------
>>> >> >>
>>> >> >
>>> >> >
>>> >>
>>> >
>>>
>>
>>
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 9:52 AM, Jake Farrell <jf...@apache.org> wrote:
> +1 to reducing this as much as possible so there is less maintenance
> overhead and setup. Ideally i would love to see all this dockerized so we
> are using a common base across the board and then either travis or jenkins
> or locally can all run those containers with the same setup in docker and
> the same outlined test scripts (which should be within the build folder) so
> this is the same repeatable process where ever it is run from
>
I said this on another thread, but I think your and other comitter limited
resources should be bent towards doing this now and not accepting patches,
making changes or doing RCs.
The thrift project feels untenable to contribute to with all the red CI.
After seeing this, I'm also growing disinclined towards depending on the
tool in my projects.
Maybe the thrift committers don't share this sense of priority, but to me a
stable green CI is the foundation of any project, without it everything
else crumbles.
>
>
>
> On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org> wrote:
>
>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>>
>> > Quoting from my previous mail.
>> >
>> > > Other than Travis, make check is hanging for almost every build of
>> > Jenkins.
>> > > The log is not that clear but I think it's D test.
>> > > AFAIK the test was running fine a few weeks ago and nobody touched it
>> > since then.
>> > > It might be due to insufficient resource on Jenkins.
>> >
>> > I suspect default task limit introduced in a recent version of docker is
>> > not lifted on ASF jenkins.
>> >
>> > I'm not sure if it's worth maintaining sub-set of builds on another CI
>> > that has relatively unstable basis that cannot even be touched by
>> > committers.
>> > Less resource is fine because it can detect failures on such platforms
>> > like last time John enabled it.
>> > But it's apparently changing.
>> >
>>
>> Aha - that would be an interesting cause to the D hangs.
>>
>> I'm not clear on what you meant by the rest, but I assume you're
>> addressing
>> the confusing fact that thrift maintains 2 sets of broken CI jobs (fwict)
>> for pull requests, TravisCI and Apache Jenkins.
>>
>> It seems to me 4 steps are needed to provide baseline sanity for
>> contributing to the project:
>> 1. Halt accepting and changes immediately.
>> 2. Pick Travis or Jenkins, kill the other.
>> 3. Get the winner from 2 green.
>> 4. Resume accepting patches that are green in CI and only green in CI.
>>
>>
>> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org>
>> wrote:
>> >
>> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>> wrote:
>> >>
>> >> >
>> >> >
>> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
>> >> wrote:
>> >> >
>> >> >> I got one build through (which failed in "d" tests) and now it's
>> stuck
>> >> in
>> >> >> the same state, see:
>> >> >> https://builds.apache.org/job/Thrift-precommit/411/
>> >> >>
>> >> >> FATAL: Could not checkout master with start point origin/master
>> >> >> hudson.plugins.git.GitException: Could not checkout master with
>> start
>> >> >> point origin/master
>> >> >> at
>> >> >>
>> >>
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>> >> >> at
>> >> >>
>> >>
>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>> >> >> at
>> >> >>
>> >>
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >> >> at
>> >> >>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >> >> at
>> >> >>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
>> >> >> at
>> >> >>
>> >>
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>> >> >> at
>> >> >>
>> >>
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>> >> >> at
>> >> >>
>> >>
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>> >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
>> >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>> >> >> at hudson.remoting.Request$2.run(Request.java:326)
>> >> >> at
>> >> >>
>> >>
>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>> >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> >> >> at
>> >> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >> >> at
>> >> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >> >> at java.lang.Thread.run(Thread.java:745)
>> >> >> at ......remote call to H10(Native Method)
>> >> >> at
>> >> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>> >> >> at
>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>> >> >> at hudson.remoting.Channel.call(Channel.java:781)
>> >> >> at
>> >> >>
>> >>
>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>> >> >> at
>> >> >>
>> >>
>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>> >> >> at
>> >> >>
>> >>
>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>> >> >> at
>> >> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
>> >> >> at
>> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>> >> >> at
>> >> >>
>> >>
>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>> >> >> at
>> >> >>
>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>> >> >> at
>> >> >>
>> >>
>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>> >> >> at hudson.model.Run.execute(Run.java:1738)
>> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>> >> >> at
>> >> >> hudson.model.ResourceController.execute(ResourceController.java:98)
>> >> >> at hudson.model.Executor.run(Executor.java:410)
>> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
>> >> >> master origin/master" returned status code 1:
>> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>> >> >>
>> >> >> stderr: error: you need to resolve your current index first
>> >> >>
>> >> >> It looks like the build environment is not forced clean at the
>> >> beginning
>> >> >> of each build.
>> >> >>
>> >> >
>> >> > Ack - looking now.
>> >> >
>> >> > It is odd that the git portion of these builds went sideways since
>> the
>> >> > Jenkins Job Config History auditing plugin shows the last change
>> >> (before my
>> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>> >> plugins
>> >> > were updated by infra causing the previously working job config to
>> not
>> >> work
>> >> > any longer.
>> >> >
>> >>
>> >> OK - that analysis was wrong, clearly there has been a change in the
>> build
>> >> itself that modifies source code and this causes the issue.
>> >> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>> >> with
>> >> the following description:
>> >>
>> >> Clean up the workspace before every checkout by deleting all untracked
>> >> files and directories, including those which are specified in
>> .gitignore.
>> >> It also resets all *tracked* files to their versioned state. This
>> ensures
>> >>
>> >> that the workspace is in the same state as if you cloned and checked
>> out
>> >> in
>> >> a brand-new empty directory, and ensures that your build is not
>> affected
>> >> by
>> >> the files generated by the previous build.
>> >>
>> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>> which
>> >> should do it. That should insulate CI from bad tests that modify
>> checked
>> >> in
>> >> repo state, but those tests shouldn't exist either.
>> >>
>> >> COMMITTERS:
>> >> I'd like to reiterate to any committers out there that red CI must be a
>> >> hard bright line that is not crossed when accepting patches; otherwise
>> >> well
>> >> be right back here after getting this thing green again. Here we is
>> you -
>> >> I won't be interested in helping out a third time if this relapses.
>> >>
>> >>
>> >> >
>> >> >> - Jim
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Jim King
>> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
>> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <jsirois@apache.org
>> >
>> >> >> Subject: RE: Build Failures
>> >> >>
>> >> >> The builds were failing claiming that a file was in the middle of
>> being
>> >> >> merged and they were all failing early.
>> >> >> I think the build environment itself is compromised and there's
>> >> nothing I
>> >> >> can do about that.
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: John Sirois [mailto:jsirois@apache.org]
>> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
>> >> >> To: dev@thrift.apache.org
>> >> >> Subject: Re: Build Failures
>> >> >>
>> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>> >> wrote:
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <
>> Jim.King@simplivity.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >> I’m still looking for answers on pull request build failures.
>> >> >> >>
>> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> >> >> >> precommit builds for strange reasons.
>> >> >> >>
>> >> >> >> The apache internal builds seem to be failing.
>> >> >> >>
>> >> >> >
>> >> >> > I think the answer is the breaks need a fixer; hopefully you can
>> find
>> >> >> > time to help fix.
>> >> >> >
>> >> >> > I say this because I started down a series of patches to the java
>> >> >> > codegen/lib a while back and found a similar state - though on the
>> >> >> > pull request builder (apache jenkins). I stopped my java stuff
>> and
>> >> >> > fixed that CI with the help of Aki and Jake reviewing and
>> providing
>> >> >> > guidance. I am not a thrift comitter.
>> >> >> >
>> >> >>
>> >> >> I will say that its discouraging that that CI is now solid red too:
>> >> >> https://builds.apache.org/job/Thrift-precommit/
>> >> >> Part of the answer IMO is for committers to hold a hard line on
>> >> accepting
>> >> >> any patch, or pushing their own, w/o full green CIs.
>> >> >>
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> >>
>> >> >> >>
>> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
>> >> >> >>
>> >> >> >> James E. King, III
>> >> >> >>
>> >> >> >> Architect
>> >> >> >>
>> >> >> >> 8 Technology Drive, 2nd Floor
>> >> >> >> Westborough, MA 01581-1756
>> >> >> >>
>> >> >> >> Ph: 855-SVT-INFO
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> ------------------------------
>> >> >> >> PRIVACY STATEMENT:
>> >> >> >> This message is a PRIVATE communication. This message and all
>> >> >> >> attachments are a private communication sent by SimpliVity and
>> are
>> >> >> >> considered to be confidential or protected by privilege. If you
>> are
>> >> >> >> not the intended recipient, you are hereby notified that any
>> >> >> >> disclosure, copying, distribution or use of the information
>> >> contained
>> >> >> >> in or attached to this message is strictly prohibited. Please
>> notify
>> >> >> >> the sender of the delivery error by replying to this message, and
>> >> then
>> >> >> delete it from your system.
>> >> >> >> For more information please visit http://www.simplivity.com
>> >> >> >> ------------------------------
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >>
>> ---------------------------------------------------------------------------------------
>> >> >> PRIVACY STATEMENT:
>> >> >> This message is a PRIVATE communication. This message and all
>> >> >> attachments are a private communication sent by SimpliVity and are
>> >> >> considered to be confidential or protected by privilege. If you are
>> >> not the
>> >> >> intended recipient, you are hereby notified that any disclosure,
>> >> copying,
>> >> >> distribution or use of the information contained in or attached to
>> this
>> >> >> message is strictly prohibited. Please notify the sender of the
>> >> delivery
>> >> >> error by replying to this message, and then delete it from your
>> system.
>> >> >>
>> >> >>
>> >>
>> ---------------------------------------------------------------------------------------
>> >> >>
>> >> >
>> >> >
>> >>
>> >
>>
>
>
Re: Build Failures
Posted by Jake Farrell <jf...@apache.org>.
+1 to reducing this as much as possible so there is less maintenance
overhead and setup. Ideally i would love to see all this dockerized so we
are using a common base across the board and then either travis or jenkins
or locally can all run those containers with the same setup in docker and
the same outlined test scripts (which should be within the build folder) so
this is the same repeatable process where ever it is run from
On Thu, Apr 14, 2016 at 11:41 AM, John Sirois <js...@apache.org> wrote:
> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>
> > Quoting from my previous mail.
> >
> > > Other than Travis, make check is hanging for almost every build of
> > Jenkins.
> > > The log is not that clear but I think it's D test.
> > > AFAIK the test was running fine a few weeks ago and nobody touched it
> > since then.
> > > It might be due to insufficient resource on Jenkins.
> >
> > I suspect default task limit introduced in a recent version of docker is
> > not lifted on ASF jenkins.
> >
> > I'm not sure if it's worth maintaining sub-set of builds on another CI
> > that has relatively unstable basis that cannot even be touched by
> > committers.
> > Less resource is fine because it can detect failures on such platforms
> > like last time John enabled it.
> > But it's apparently changing.
> >
>
> Aha - that would be an interesting cause to the D hangs.
>
> I'm not clear on what you meant by the rest, but I assume you're addressing
> the confusing fact that thrift maintains 2 sets of broken CI jobs (fwict)
> for pull requests, TravisCI and Apache Jenkins.
>
> It seems to me 4 steps are needed to provide baseline sanity for
> contributing to the project:
> 1. Halt accepting and changes immediately.
> 2. Pick Travis or Jenkins, kill the other.
> 3. Get the winner from 2 green.
> 4. Resume accepting patches that are green in CI and only green in CI.
>
>
> > On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org> wrote:
> >
> >> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
> wrote:
> >>
> >> >
> >> >
> >> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
> >> wrote:
> >> >
> >> >> I got one build through (which failed in "d" tests) and now it's
> stuck
> >> in
> >> >> the same state, see:
> >> >> https://builds.apache.org/job/Thrift-precommit/411/
> >> >>
> >> >> FATAL: Could not checkout master with start point origin/master
> >> >> hudson.plugins.git.GitException: Could not checkout master with start
> >> >> point origin/master
> >> >> at
> >> >>
> >>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
> >> >> at
> >> >>
> >>
> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
> >> >> at
> >> >>
> >>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >> >> at
> >> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> >> at
> >> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> >> at
> >> >>
> >>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
> >> >> at
> >> >>
> >>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
> >> >> at
> >> >>
> >>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
> >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
> >> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> >> >> at hudson.remoting.Request$2.run(Request.java:326)
> >> >> at
> >> >>
> >>
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
> >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> >> at
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> >> at
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> >> at java.lang.Thread.run(Thread.java:745)
> >> >> at ......remote call to H10(Native Method)
> >> >> at
> >> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
> >> >> at
> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
> >> >> at hudson.remoting.Channel.call(Channel.java:781)
> >> >> at
> >> >>
> >>
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
> >> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
> >> >> at
> >> >>
> >>
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
> >> >> at
> >> >>
> >>
> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
> >> >> at
> >> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
> >> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
> >> >> at hudson.scm.SCM.checkout(SCM.java:485)
> >> >> at
> >> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
> >> >> at
> >> >>
> >>
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
> >> >> at
> >> >> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >> >> at
> >> >>
> >>
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
> >> >> at hudson.model.Run.execute(Run.java:1738)
> >> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >> >> at
> >> >> hudson.model.ResourceController.execute(ResourceController.java:98)
> >> >> at hudson.model.Executor.run(Executor.java:410)
> >> >> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
> >> >> master origin/master" returned status code 1:
> >> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
> >> >>
> >> >> stderr: error: you need to resolve your current index first
> >> >>
> >> >> It looks like the build environment is not forced clean at the
> >> beginning
> >> >> of each build.
> >> >>
> >> >
> >> > Ack - looking now.
> >> >
> >> > It is odd that the git portion of these builds went sideways since the
> >> > Jenkins Job Config History auditing plugin shows the last change
> >> (before my
> >> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
> >> plugins
> >> > were updated by infra causing the previously working job config to not
> >> work
> >> > any longer.
> >> >
> >>
> >> OK - that analysis was wrong, clearly there has been a change in the
> build
> >> itself that modifies source code and this causes the issue.
> >> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
> >> with
> >> the following description:
> >>
> >> Clean up the workspace before every checkout by deleting all untracked
> >> files and directories, including those which are specified in
> .gitignore.
> >> It also resets all *tracked* files to their versioned state. This
> ensures
> >>
> >> that the workspace is in the same state as if you cloned and checked out
> >> in
> >> a brand-new empty directory, and ensures that your build is not affected
> >> by
> >> the files generated by the previous build.
> >>
> >> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
> which
> >> should do it. That should insulate CI from bad tests that modify checked
> >> in
> >> repo state, but those tests shouldn't exist either.
> >>
> >> COMMITTERS:
> >> I'd like to reiterate to any committers out there that red CI must be a
> >> hard bright line that is not crossed when accepting patches; otherwise
> >> well
> >> be right back here after getting this thing green again. Here we is
> you -
> >> I won't be interested in helping out a third time if this relapses.
> >>
> >>
> >> >
> >> >> - Jim
> >> >>
> >> >> -----Original Message-----
> >> >> From: Jim King
> >> >> Sent: Wednesday, April 13, 2016 10:34 PM
> >> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
> >> >> Subject: RE: Build Failures
> >> >>
> >> >> The builds were failing claiming that a file was in the middle of
> being
> >> >> merged and they were all failing early.
> >> >> I think the build environment itself is compromised and there's
> >> nothing I
> >> >> can do about that.
> >> >>
> >> >> -----Original Message-----
> >> >> From: John Sirois [mailto:jsirois@apache.org]
> >> >> Sent: Wednesday, April 13, 2016 9:58 PM
> >> >> To: dev@thrift.apache.org
> >> >> Subject: Re: Build Failures
> >> >>
> >> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
> >> wrote:
> >> >>
> >> >> >
> >> >> >
> >> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Jim.King@simplivity.com
> >
> >> >> wrote:
> >> >> >
> >> >> >> I’m still looking for answers on pull request build failures.
> >> >> >>
> >> >> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
> >> >> >> precommit builds for strange reasons.
> >> >> >>
> >> >> >> The apache internal builds seem to be failing.
> >> >> >>
> >> >> >
> >> >> > I think the answer is the breaks need a fixer; hopefully you can
> find
> >> >> > time to help fix.
> >> >> >
> >> >> > I say this because I started down a series of patches to the java
> >> >> > codegen/lib a while back and found a similar state - though on the
> >> >> > pull request builder (apache jenkins). I stopped my java stuff and
> >> >> > fixed that CI with the help of Aki and Jake reviewing and providing
> >> >> > guidance. I am not a thrift comitter.
> >> >> >
> >> >>
> >> >> I will say that its discouraging that that CI is now solid red too:
> >> >> https://builds.apache.org/job/Thrift-precommit/
> >> >> Part of the answer IMO is for committers to hold a hard line on
> >> accepting
> >> >> any patch, or pushing their own, w/o full green CIs.
> >> >>
> >> >>
> >> >> >
> >> >> >
> >> >> >>
> >> >> >>
> >> >> >> [image: Description: Description: simplivity-lg-xsmall]
> >> >> >>
> >> >> >> James E. King, III
> >> >> >>
> >> >> >> Architect
> >> >> >>
> >> >> >> 8 Technology Drive, 2nd Floor
> >> >> >> Westborough, MA 01581-1756
> >> >> >>
> >> >> >> Ph: 855-SVT-INFO
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> ------------------------------
> >> >> >> PRIVACY STATEMENT:
> >> >> >> This message is a PRIVATE communication. This message and all
> >> >> >> attachments are a private communication sent by SimpliVity and are
> >> >> >> considered to be confidential or protected by privilege. If you
> are
> >> >> >> not the intended recipient, you are hereby notified that any
> >> >> >> disclosure, copying, distribution or use of the information
> >> contained
> >> >> >> in or attached to this message is strictly prohibited. Please
> notify
> >> >> >> the sender of the delivery error by replying to this message, and
> >> then
> >> >> delete it from your system.
> >> >> >> For more information please visit http://www.simplivity.com
> >> >> >> ------------------------------
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >>
> >>
> ---------------------------------------------------------------------------------------
> >> >> PRIVACY STATEMENT:
> >> >> This message is a PRIVATE communication. This message and all
> >> >> attachments are a private communication sent by SimpliVity and are
> >> >> considered to be confidential or protected by privilege. If you are
> >> not the
> >> >> intended recipient, you are hereby notified that any disclosure,
> >> copying,
> >> >> distribution or use of the information contained in or attached to
> this
> >> >> message is strictly prohibited. Please notify the sender of the
> >> delivery
> >> >> error by replying to this message, and then delete it from your
> system.
> >> >>
> >> >>
> >>
> ---------------------------------------------------------------------------------------
> >> >>
> >> >
> >> >
> >>
> >
>
Re: Build Failures
Posted by Aki Sukegawa <ns...@apache.org>.
Jake, everything is dockerized on both Jenkins and Travis.
The current Jenkins failure (D test hang) is caused by different
environment on Jenkins.
On Fri, Apr 15, 2016 at 12:45 AM John Sirois <js...@apache.org> wrote:
> On Thu, Apr 14, 2016 at 9:41 AM, John Sirois <js...@apache.org> wrote:
>
>>
>>
>> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>>
>>> Quoting from my previous mail.
>>>
>>> > Other than Travis, make check is hanging for almost every build of
>>> Jenkins.
>>> > The log is not that clear but I think it's D test.
>>> > AFAIK the test was running fine a few weeks ago and nobody touched it
>>> since then.
>>> > It might be due to insufficient resource on Jenkins.
>>>
>>> I suspect default task limit introduced in a recent version of docker is
>>> not lifted on ASF jenkins.
>>>
>>> I'm not sure if it's worth maintaining sub-set of builds on another CI
>>> that has relatively unstable basis that cannot even be touched by
>>> committers.
>>> Less resource is fine because it can detect failures on such platforms
>>> like last time John enabled it.
>>> But it's apparently changing.
>>>
>>
>> Aha - that would be an interesting cause to the D hangs.
>>
>> I'm not clear on what you meant by the rest, but I assume you're
>> addressing the confusing fact that thrift maintains 2 sets of broken CI
>> jobs (fwict) for pull requests, TravisCI and Apache Jenkins.
>>
>> It seems to me 4 steps are needed to provide baseline sanity for
>> contributing to the project:
>> 1. Halt accepting and changes immediately.
>> 2. Pick Travis or Jenkins, kill the other.
>> 3. Get the winner from 2 green.
>> 4. Resume accepting patches that are green in CI and only green in CI.
>>
>
> Towards step 2, I think I now have the git issue solved for Jenkins after
> enabling `git clean -fdx && git reset --hard HEAD` (or the equivalent in
> the Jenkins git plugin) and modifying the `docker run`s to use --user `uid
> -u`:`uid -g` so that the container modifications to the Jenkins WORKSPACE
> volume mount are done as the Jenkins user instead of as root.
> https://builds.apache.org/job/Thrift-precommit/417/ is spinning with
> these fixes and hopefully goes clean to a hang in the D tests.
>
>
>>
>>> On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org> wrote:
>>>
>>>> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org>
>>>> wrote:
>>>>
>>>> >
>>>> >
>>>> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
>>>> wrote:
>>>> >
>>>> >> I got one build through (which failed in "d" tests) and now it's
>>>> stuck in
>>>> >> the same state, see:
>>>> >> https://builds.apache.org/job/Thrift-precommit/411/
>>>> >>
>>>> >> FATAL: Could not checkout master with start point origin/master
>>>> >> hudson.plugins.git.GitException: Could not checkout master with start
>>>> >> point origin/master
>>>> >> at
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>>> >> at
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>>> >> at
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> >> at
>>>> >>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> >> at
>>>> >>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> >> at
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>>> >> at
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>>> >> at
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>>> >> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>>> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>>> >> at hudson.remoting.Request$2.run(Request.java:326)
>>>> >> at
>>>> >>
>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> >> at
>>>> >>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> >> at
>>>> >>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> >> at java.lang.Thread.run(Thread.java:745)
>>>> >> at ......remote call to H10(Native Method)
>>>> >> at
>>>> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>>> >> at
>>>> hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>>> >> at hudson.remoting.Channel.call(Channel.java:781)
>>>> >> at
>>>> >>
>>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>>> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>>> >> at
>>>> >>
>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>>> >> at
>>>> >>
>>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>>> >> at
>>>> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>>> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>>> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>>> >> at
>>>> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>>> >> at
>>>> >>
>>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>>> >> at
>>>> >> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>>> >> at
>>>> >>
>>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>>> >> at hudson.model.Run.execute(Run.java:1738)
>>>> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>>> >> at
>>>> >> hudson.model.ResourceController.execute(ResourceController.java:98)
>>>> >> at hudson.model.Executor.run(Executor.java:410)
>>>> >> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
>>>> >> master origin/master" returned status code 1:
>>>> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>>> >>
>>>> >> stderr: error: you need to resolve your current index first
>>>> >>
>>>> >> It looks like the build environment is not forced clean at the
>>>> beginning
>>>> >> of each build.
>>>> >>
>>>> >
>>>> > Ack - looking now.
>>>> >
>>>> > It is odd that the git portion of these builds went sideways since the
>>>> > Jenkins Job Config History auditing plugin shows the last change
>>>> (before my
>>>> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>>>> plugins
>>>> > were updated by infra causing the previously working job config to
>>>> not work
>>>> > any longer.
>>>> >
>>>>
>>>> OK - that analysis was wrong, clearly there has been a change in the
>>>> build
>>>> itself that modifies source code and this causes the issue.
>>>> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>>> with
>>>> the following description:
>>>>
>>>> Clean up the workspace before every checkout by deleting all untracked
>>>> files and directories, including those which are specified in
>>>> .gitignore.
>>>> It also resets all *tracked* files to their versioned state. This
>>>> ensures
>>>>
>>>> that the workspace is in the same state as if you cloned and checked
>>>> out in
>>>> a brand-new empty directory, and ensures that your build is not
>>>> affected by
>>>> the files generated by the previous build.
>>>>
>>>> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me,
>>>> which
>>>> should do it. That should insulate CI from bad tests that modify
>>>> checked in
>>>> repo state, but those tests shouldn't exist either.
>>>>
>>>> COMMITTERS:
>>>> I'd like to reiterate to any committers out there that red CI must be a
>>>> hard bright line that is not crossed when accepting patches; otherwise
>>>> well
>>>> be right back here after getting this thing green again. Here we is
>>>> you -
>>>> I won't be interested in helping out a third time if this relapses.
>>>>
>>>>
>>>> >
>>>> >> - Jim
>>>> >>
>>>> >> -----Original Message-----
>>>> >> From: Jim King
>>>> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>>> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
>>>> >> Subject: RE: Build Failures
>>>> >>
>>>> >> The builds were failing claiming that a file was in the middle of
>>>> being
>>>> >> merged and they were all failing early.
>>>> >> I think the build environment itself is compromised and there's
>>>> nothing I
>>>> >> can do about that.
>>>> >>
>>>> >> -----Original Message-----
>>>> >> From: John Sirois [mailto:jsirois@apache.org]
>>>> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>>> >> To: dev@thrift.apache.org
>>>> >> Subject: Re: Build Failures
>>>> >>
>>>> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>>>> wrote:
>>>> >>
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Jim.King@simplivity.com
>>>> >
>>>> >> wrote:
>>>> >> >
>>>> >> >> I’m still looking for answers on pull request build failures.
>>>> >> >>
>>>> >> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
>>>> >> >> precommit builds for strange reasons.
>>>> >> >>
>>>> >> >> The apache internal builds seem to be failing.
>>>> >> >>
>>>> >> >
>>>> >> > I think the answer is the breaks need a fixer; hopefully you can
>>>> find
>>>> >> > time to help fix.
>>>> >> >
>>>> >> > I say this because I started down a series of patches to the java
>>>> >> > codegen/lib a while back and found a similar state - though on the
>>>> >> > pull request builder (apache jenkins). I stopped my java stuff and
>>>> >> > fixed that CI with the help of Aki and Jake reviewing and providing
>>>> >> > guidance. I am not a thrift comitter.
>>>> >> >
>>>> >>
>>>> >> I will say that its discouraging that that CI is now solid red too:
>>>> >> https://builds.apache.org/job/Thrift-precommit/
>>>> >> Part of the answer IMO is for committers to hold a hard line on
>>>> accepting
>>>> >> any patch, or pushing their own, w/o full green CIs.
>>>> >>
>>>> >>
>>>> >> >
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>>> >> >>
>>>> >> >> James E. King, III
>>>> >> >>
>>>> >> >> Architect
>>>> >> >>
>>>> >> >> 8 Technology Drive, 2nd Floor
>>>> >> >> Westborough, MA 01581-1756
>>>> >> >>
>>>> >> >> Ph: 855-SVT-INFO
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> ------------------------------
>>>> >> >> PRIVACY STATEMENT:
>>>> >> >> This message is a PRIVATE communication. This message and all
>>>> >> >> attachments are a private communication sent by SimpliVity and are
>>>> >> >> considered to be confidential or protected by privilege. If you
>>>> are
>>>> >> >> not the intended recipient, you are hereby notified that any
>>>> >> >> disclosure, copying, distribution or use of the information
>>>> contained
>>>> >> >> in or attached to this message is strictly prohibited. Please
>>>> notify
>>>> >> >> the sender of the delivery error by replying to this message, and
>>>> then
>>>> >> delete it from your system.
>>>> >> >> For more information please visit http://www.simplivity.com
>>>> >> >> ------------------------------
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >>
>>>> >>
>>>> ---------------------------------------------------------------------------------------
>>>> >> PRIVACY STATEMENT:
>>>> >> This message is a PRIVATE communication. This message and all
>>>> >> attachments are a private communication sent by SimpliVity and are
>>>> >> considered to be confidential or protected by privilege. If you are
>>>> not the
>>>> >> intended recipient, you are hereby notified that any disclosure,
>>>> copying,
>>>> >> distribution or use of the information contained in or attached to
>>>> this
>>>> >> message is strictly prohibited. Please notify the sender of the
>>>> delivery
>>>> >> error by replying to this message, and then delete it from your
>>>> system.
>>>> >>
>>>> >>
>>>> ---------------------------------------------------------------------------------------
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 9:41 AM, John Sirois <js...@apache.org> wrote:
>
>
> On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
>
>> Quoting from my previous mail.
>>
>> > Other than Travis, make check is hanging for almost every build of
>> Jenkins.
>> > The log is not that clear but I think it's D test.
>> > AFAIK the test was running fine a few weeks ago and nobody touched it
>> since then.
>> > It might be due to insufficient resource on Jenkins.
>>
>> I suspect default task limit introduced in a recent version of docker is
>> not lifted on ASF jenkins.
>>
>> I'm not sure if it's worth maintaining sub-set of builds on another CI
>> that has relatively unstable basis that cannot even be touched by
>> committers.
>> Less resource is fine because it can detect failures on such platforms
>> like last time John enabled it.
>> But it's apparently changing.
>>
>
> Aha - that would be an interesting cause to the D hangs.
>
> I'm not clear on what you meant by the rest, but I assume you're
> addressing the confusing fact that thrift maintains 2 sets of broken CI
> jobs (fwict) for pull requests, TravisCI and Apache Jenkins.
>
> It seems to me 4 steps are needed to provide baseline sanity for
> contributing to the project:
> 1. Halt accepting and changes immediately.
> 2. Pick Travis or Jenkins, kill the other.
> 3. Get the winner from 2 green.
> 4. Resume accepting patches that are green in CI and only green in CI.
>
Towards step 2, I think I now have the git issue solved for Jenkins after
enabling `git clean -fdx && git reset --hard HEAD` (or the equivalent in
the Jenkins git plugin) and modifying the `docker run`s to use --user `uid
-u`:`uid -g` so that the container modifications to the Jenkins WORKSPACE
volume mount are done as the Jenkins user instead of as root.
https://builds.apache.org/job/Thrift-precommit/417/ is spinning with these
fixes and hopefully goes clean to a hang in the D tests.
>
>> On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org> wrote:
>>
>>> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org> wrote:
>>>
>>> >
>>> >
>>> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
>>> wrote:
>>> >
>>> >> I got one build through (which failed in "d" tests) and now it's
>>> stuck in
>>> >> the same state, see:
>>> >> https://builds.apache.org/job/Thrift-precommit/411/
>>> >>
>>> >> FATAL: Could not checkout master with start point origin/master
>>> >> hudson.plugins.git.GitException: Could not checkout master with start
>>> >> point origin/master
>>> >> at
>>> >>
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>>> >> at
>>> >>
>>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>>> >> at
>>> >>
>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> >> at
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> >> at
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >> at java.lang.reflect.Method.invoke(Method.java:606)
>>> >> at
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>>> >> at
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>>> >> at
>>> >>
>>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>>> >> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
>>> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>>> >> at hudson.remoting.Request$2.run(Request.java:326)
>>> >> at
>>> >>
>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> >> at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >> at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >> at java.lang.Thread.run(Thread.java:745)
>>> >> at ......remote call to H10(Native Method)
>>> >> at
>>> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>>> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>>> >> at hudson.remoting.Channel.call(Channel.java:781)
>>> >> at
>>> >>
>>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>>> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>>> >> at
>>> >>
>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>>> >> at
>>> >>
>>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>>> >> at
>>> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>>> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>>> >> at hudson.scm.SCM.checkout(SCM.java:485)
>>> >> at
>>> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>>> >> at
>>> >>
>>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>>> >> at
>>> >> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>>> >> at
>>> >>
>>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>>> >> at hudson.model.Run.execute(Run.java:1738)
>>> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>> >> at
>>> >> hudson.model.ResourceController.execute(ResourceController.java:98)
>>> >> at hudson.model.Executor.run(Executor.java:410)
>>> >> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
>>> >> master origin/master" returned status code 1:
>>> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>> >>
>>> >> stderr: error: you need to resolve your current index first
>>> >>
>>> >> It looks like the build environment is not forced clean at the
>>> beginning
>>> >> of each build.
>>> >>
>>> >
>>> > Ack - looking now.
>>> >
>>> > It is odd that the git portion of these builds went sideways since the
>>> > Jenkins Job Config History auditing plugin shows the last change
>>> (before my
>>> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>>> plugins
>>> > were updated by infra causing the previously working job config to not
>>> work
>>> > any longer.
>>> >
>>>
>>> OK - that analysis was wrong, clearly there has been a change in the
>>> build
>>> itself that modifies source code and this causes the issue.
>>> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>>> with
>>> the following description:
>>>
>>> Clean up the workspace before every checkout by deleting all untracked
>>> files and directories, including those which are specified in .gitignore.
>>> It also resets all *tracked* files to their versioned state. This ensures
>>>
>>> that the workspace is in the same state as if you cloned and checked out
>>> in
>>> a brand-new empty directory, and ensures that your build is not affected
>>> by
>>> the files generated by the previous build.
>>>
>>> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me, which
>>> should do it. That should insulate CI from bad tests that modify checked
>>> in
>>> repo state, but those tests shouldn't exist either.
>>>
>>> COMMITTERS:
>>> I'd like to reiterate to any committers out there that red CI must be a
>>> hard bright line that is not crossed when accepting patches; otherwise
>>> well
>>> be right back here after getting this thing green again. Here we is you
>>> -
>>> I won't be interested in helping out a third time if this relapses.
>>>
>>>
>>> >
>>> >> - Jim
>>> >>
>>> >> -----Original Message-----
>>> >> From: Jim King
>>> >> Sent: Wednesday, April 13, 2016 10:34 PM
>>> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
>>> >> Subject: RE: Build Failures
>>> >>
>>> >> The builds were failing claiming that a file was in the middle of
>>> being
>>> >> merged and they were all failing early.
>>> >> I think the build environment itself is compromised and there's
>>> nothing I
>>> >> can do about that.
>>> >>
>>> >> -----Original Message-----
>>> >> From: John Sirois [mailto:jsirois@apache.org]
>>> >> Sent: Wednesday, April 13, 2016 9:58 PM
>>> >> To: dev@thrift.apache.org
>>> >> Subject: Re: Build Failures
>>> >>
>>> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>>> wrote:
>>> >>
>>> >> >
>>> >> >
>>> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
>>> >> wrote:
>>> >> >
>>> >> >> I’m still looking for answers on pull request build failures.
>>> >> >>
>>> >> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
>>> >> >> precommit builds for strange reasons.
>>> >> >>
>>> >> >> The apache internal builds seem to be failing.
>>> >> >>
>>> >> >
>>> >> > I think the answer is the breaks need a fixer; hopefully you can
>>> find
>>> >> > time to help fix.
>>> >> >
>>> >> > I say this because I started down a series of patches to the java
>>> >> > codegen/lib a while back and found a similar state - though on the
>>> >> > pull request builder (apache jenkins). I stopped my java stuff and
>>> >> > fixed that CI with the help of Aki and Jake reviewing and providing
>>> >> > guidance. I am not a thrift comitter.
>>> >> >
>>> >>
>>> >> I will say that its discouraging that that CI is now solid red too:
>>> >> https://builds.apache.org/job/Thrift-precommit/
>>> >> Part of the answer IMO is for committers to hold a hard line on
>>> accepting
>>> >> any patch, or pushing their own, w/o full green CIs.
>>> >>
>>> >>
>>> >> >
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> [image: Description: Description: simplivity-lg-xsmall]
>>> >> >>
>>> >> >> James E. King, III
>>> >> >>
>>> >> >> Architect
>>> >> >>
>>> >> >> 8 Technology Drive, 2nd Floor
>>> >> >> Westborough, MA 01581-1756
>>> >> >>
>>> >> >> Ph: 855-SVT-INFO
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> ------------------------------
>>> >> >> PRIVACY STATEMENT:
>>> >> >> This message is a PRIVATE communication. This message and all
>>> >> >> attachments are a private communication sent by SimpliVity and are
>>> >> >> considered to be confidential or protected by privilege. If you are
>>> >> >> not the intended recipient, you are hereby notified that any
>>> >> >> disclosure, copying, distribution or use of the information
>>> contained
>>> >> >> in or attached to this message is strictly prohibited. Please
>>> notify
>>> >> >> the sender of the delivery error by replying to this message, and
>>> then
>>> >> delete it from your system.
>>> >> >> For more information please visit http://www.simplivity.com
>>> >> >> ------------------------------
>>> >> >>
>>> >> >
>>> >> >
>>> >>
>>> >>
>>> ---------------------------------------------------------------------------------------
>>> >> PRIVACY STATEMENT:
>>> >> This message is a PRIVATE communication. This message and all
>>> >> attachments are a private communication sent by SimpliVity and are
>>> >> considered to be confidential or protected by privilege. If you are
>>> not the
>>> >> intended recipient, you are hereby notified that any disclosure,
>>> copying,
>>> >> distribution or use of the information contained in or attached to
>>> this
>>> >> message is strictly prohibited. Please notify the sender of the
>>> delivery
>>> >> error by replying to this message, and then delete it from your
>>> system.
>>> >>
>>> >>
>>> ---------------------------------------------------------------------------------------
>>> >>
>>> >
>>> >
>>>
>>
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 9:34 AM, Aki Sukegawa <ns...@apache.org> wrote:
> Quoting from my previous mail.
>
> > Other than Travis, make check is hanging for almost every build of
> Jenkins.
> > The log is not that clear but I think it's D test.
> > AFAIK the test was running fine a few weeks ago and nobody touched it
> since then.
> > It might be due to insufficient resource on Jenkins.
>
> I suspect default task limit introduced in a recent version of docker is
> not lifted on ASF jenkins.
>
> I'm not sure if it's worth maintaining sub-set of builds on another CI
> that has relatively unstable basis that cannot even be touched by
> committers.
> Less resource is fine because it can detect failures on such platforms
> like last time John enabled it.
> But it's apparently changing.
>
Aha - that would be an interesting cause to the D hangs.
I'm not clear on what you meant by the rest, but I assume you're addressing
the confusing fact that thrift maintains 2 sets of broken CI jobs (fwict)
for pull requests, TravisCI and Apache Jenkins.
It seems to me 4 steps are needed to provide baseline sanity for
contributing to the project:
1. Halt accepting and changes immediately.
2. Pick Travis or Jenkins, kill the other.
3. Get the winner from 2 green.
4. Resume accepting patches that are green in CI and only green in CI.
> On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org> wrote:
>
>> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org> wrote:
>>
>> >
>> >
>> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
>> wrote:
>> >
>> >> I got one build through (which failed in "d" tests) and now it's stuck
>> in
>> >> the same state, see:
>> >> https://builds.apache.org/job/Thrift-precommit/411/
>> >>
>> >> FATAL: Could not checkout master with start point origin/master
>> >> hudson.plugins.git.GitException: Could not checkout master with start
>> >> point origin/master
>> >> at
>> >>
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>> >> at
>> >>
>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>> >> at
>> >>
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >> at
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:606)
>> >> at
>> >>
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>> >> at
>> >>
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>> >> at
>> >>
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>> >> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
>> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>> >> at hudson.remoting.Request$2.run(Request.java:326)
>> >> at
>> >>
>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> >> at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >> at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >> at ......remote call to H10(Native Method)
>> >> at
>> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>> >> at hudson.remoting.Channel.call(Channel.java:781)
>> >> at
>> >>
>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>> >> at
>> >>
>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>> >> at
>> >>
>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>> >> at
>> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>> >> at hudson.scm.SCM.checkout(SCM.java:485)
>> >> at
>> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>> >> at
>> >>
>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>> >> at
>> >> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>> >> at
>> >>
>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>> >> at hudson.model.Run.execute(Run.java:1738)
>> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>> >> at
>> >> hudson.model.ResourceController.execute(ResourceController.java:98)
>> >> at hudson.model.Executor.run(Executor.java:410)
>> >> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
>> >> master origin/master" returned status code 1:
>> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
>> >>
>> >> stderr: error: you need to resolve your current index first
>> >>
>> >> It looks like the build environment is not forced clean at the
>> beginning
>> >> of each build.
>> >>
>> >
>> > Ack - looking now.
>> >
>> > It is odd that the git portion of these builds went sideways since the
>> > Jenkins Job Config History auditing plugin shows the last change
>> (before my
>> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
>> plugins
>> > were updated by infra causing the previously working job config to not
>> work
>> > any longer.
>> >
>>
>> OK - that analysis was wrong, clearly there has been a change in the build
>> itself that modifies source code and this causes the issue.
>> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/>
>> with
>> the following description:
>>
>> Clean up the workspace before every checkout by deleting all untracked
>> files and directories, including those which are specified in .gitignore.
>> It also resets all *tracked* files to their versioned state. This ensures
>>
>> that the workspace is in the same state as if you cloned and checked out
>> in
>> a brand-new empty directory, and ensures that your build is not affected
>> by
>> the files generated by the previous build.
>>
>> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me, which
>> should do it. That should insulate CI from bad tests that modify checked
>> in
>> repo state, but those tests shouldn't exist either.
>>
>> COMMITTERS:
>> I'd like to reiterate to any committers out there that red CI must be a
>> hard bright line that is not crossed when accepting patches; otherwise
>> well
>> be right back here after getting this thing green again. Here we is you -
>> I won't be interested in helping out a third time if this relapses.
>>
>>
>> >
>> >> - Jim
>> >>
>> >> -----Original Message-----
>> >> From: Jim King
>> >> Sent: Wednesday, April 13, 2016 10:34 PM
>> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
>> >> Subject: RE: Build Failures
>> >>
>> >> The builds were failing claiming that a file was in the middle of being
>> >> merged and they were all failing early.
>> >> I think the build environment itself is compromised and there's
>> nothing I
>> >> can do about that.
>> >>
>> >> -----Original Message-----
>> >> From: John Sirois [mailto:jsirois@apache.org]
>> >> Sent: Wednesday, April 13, 2016 9:58 PM
>> >> To: dev@thrift.apache.org
>> >> Subject: Re: Build Failures
>> >>
>> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
>> wrote:
>> >>
>> >> >
>> >> >
>> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
>> >> wrote:
>> >> >
>> >> >> I’m still looking for answers on pull request build failures.
>> >> >>
>> >> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> >> >> precommit builds for strange reasons.
>> >> >>
>> >> >> The apache internal builds seem to be failing.
>> >> >>
>> >> >
>> >> > I think the answer is the breaks need a fixer; hopefully you can find
>> >> > time to help fix.
>> >> >
>> >> > I say this because I started down a series of patches to the java
>> >> > codegen/lib a while back and found a similar state - though on the
>> >> > pull request builder (apache jenkins). I stopped my java stuff and
>> >> > fixed that CI with the help of Aki and Jake reviewing and providing
>> >> > guidance. I am not a thrift comitter.
>> >> >
>> >>
>> >> I will say that its discouraging that that CI is now solid red too:
>> >> https://builds.apache.org/job/Thrift-precommit/
>> >> Part of the answer IMO is for committers to hold a hard line on
>> accepting
>> >> any patch, or pushing their own, w/o full green CIs.
>> >>
>> >>
>> >> >
>> >> >
>> >> >>
>> >> >>
>> >> >> [image: Description: Description: simplivity-lg-xsmall]
>> >> >>
>> >> >> James E. King, III
>> >> >>
>> >> >> Architect
>> >> >>
>> >> >> 8 Technology Drive, 2nd Floor
>> >> >> Westborough, MA 01581-1756
>> >> >>
>> >> >> Ph: 855-SVT-INFO
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------
>> >> >> PRIVACY STATEMENT:
>> >> >> This message is a PRIVATE communication. This message and all
>> >> >> attachments are a private communication sent by SimpliVity and are
>> >> >> considered to be confidential or protected by privilege. If you are
>> >> >> not the intended recipient, you are hereby notified that any
>> >> >> disclosure, copying, distribution or use of the information
>> contained
>> >> >> in or attached to this message is strictly prohibited. Please notify
>> >> >> the sender of the delivery error by replying to this message, and
>> then
>> >> delete it from your system.
>> >> >> For more information please visit http://www.simplivity.com
>> >> >> ------------------------------
>> >> >>
>> >> >
>> >> >
>> >>
>> >>
>> ---------------------------------------------------------------------------------------
>> >> PRIVACY STATEMENT:
>> >> This message is a PRIVATE communication. This message and all
>> >> attachments are a private communication sent by SimpliVity and are
>> >> considered to be confidential or protected by privilege. If you are
>> not the
>> >> intended recipient, you are hereby notified that any disclosure,
>> copying,
>> >> distribution or use of the information contained in or attached to this
>> >> message is strictly prohibited. Please notify the sender of the
>> delivery
>> >> error by replying to this message, and then delete it from your system.
>> >>
>> >>
>> ---------------------------------------------------------------------------------------
>> >>
>> >
>> >
>>
>
Re: Build Failures
Posted by Aki Sukegawa <ns...@apache.org>.
Quoting from my previous mail.
> Other than Travis, make check is hanging for almost every build of
Jenkins.
> The log is not that clear but I think it's D test.
> AFAIK the test was running fine a few weeks ago and nobody touched it
since then.
> It might be due to insufficient resource on Jenkins.
I suspect default task limit introduced in a recent version of docker is
not lifted on ASF jenkins.
I'm not sure if it's worth maintaining sub-set of builds on another CI that
has relatively unstable basis that cannot even be touched by committers.
Less resource is fine because it can detect failures on such platforms like
last time John enabled it.
But it's apparently changing.
On Thu, Apr 14, 2016 at 11:45 PM John Sirois <js...@apache.org> wrote:
> On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org> wrote:
>
> >
> >
> > On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com>
> wrote:
> >
> >> I got one build through (which failed in "d" tests) and now it's stuck
> in
> >> the same state, see:
> >> https://builds.apache.org/job/Thrift-precommit/411/
> >>
> >> FATAL: Could not checkout master with start point origin/master
> >> hudson.plugins.git.GitException: Could not checkout master with start
> >> point origin/master
> >> at
> >>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
> >> at
> >>
> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
> >> at
> >>
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at
> >>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
> >> at
> >>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
> >> at
> >>
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
> >> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
> >> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> >> at hudson.remoting.Request$2.run(Request.java:326)
> >> at
> >>
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >> at ......remote call to H10(Native Method)
> >> at
> >> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
> >> at hudson.remoting.Channel.call(Channel.java:781)
> >> at
> >>
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
> >> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
> >> at
> >>
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
> >> at
> >>
> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
> >> at
> >> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
> >> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
> >> at hudson.scm.SCM.checkout(SCM.java:485)
> >> at
> >> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
> >> at
> >>
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
> >> at
> >> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >> at
> >>
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
> >> at hudson.model.Run.execute(Run.java:1738)
> >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >> at
> >> hudson.model.ResourceController.execute(ResourceController.java:98)
> >> at hudson.model.Executor.run(Executor.java:410)
> >> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
> >> master origin/master" returned status code 1:
> >> stdout: lib/lua/TCompactProtocol.lua: needs merge
> >>
> >> stderr: error: you need to resolve your current index first
> >>
> >> It looks like the build environment is not forced clean at the beginning
> >> of each build.
> >>
> >
> > Ack - looking now.
> >
> > It is odd that the git portion of these builds went sideways since the
> > Jenkins Job Config History auditing plugin shows the last change (before
> my
> > tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its
> plugins
> > were updated by infra causing the previously working job config to not
> work
> > any longer.
> >
>
> OK - that analysis was wrong, clearly there has been a change in the build
> itself that modifies source code and this causes the issue.
> I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/> with
> the following description:
>
> Clean up the workspace before every checkout by deleting all untracked
> files and directories, including those which are specified in .gitignore.
> It also resets all *tracked* files to their versioned state. This ensures
> that the workspace is in the same state as if you cloned and checked out in
> a brand-new empty directory, and ensures that your build is not affected by
> the files generated by the previous build.
>
> That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me, which
> should do it. That should insulate CI from bad tests that modify checked in
> repo state, but those tests shouldn't exist either.
>
> COMMITTERS:
> I'd like to reiterate to any committers out there that red CI must be a
> hard bright line that is not crossed when accepting patches; otherwise well
> be right back here after getting this thing green again. Here we is you -
> I won't be interested in helping out a third time if this relapses.
>
>
> >
> >> - Jim
> >>
> >> -----Original Message-----
> >> From: Jim King
> >> Sent: Wednesday, April 13, 2016 10:34 PM
> >> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
> >> Subject: RE: Build Failures
> >>
> >> The builds were failing claiming that a file was in the middle of being
> >> merged and they were all failing early.
> >> I think the build environment itself is compromised and there's nothing
> I
> >> can do about that.
> >>
> >> -----Original Message-----
> >> From: John Sirois [mailto:jsirois@apache.org]
> >> Sent: Wednesday, April 13, 2016 9:58 PM
> >> To: dev@thrift.apache.org
> >> Subject: Re: Build Failures
> >>
> >> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org>
> wrote:
> >>
> >> >
> >> >
> >> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
> >> wrote:
> >> >
> >> >> I’m still looking for answers on pull request build failures.
> >> >>
> >> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
> >> >> precommit builds for strange reasons.
> >> >>
> >> >> The apache internal builds seem to be failing.
> >> >>
> >> >
> >> > I think the answer is the breaks need a fixer; hopefully you can find
> >> > time to help fix.
> >> >
> >> > I say this because I started down a series of patches to the java
> >> > codegen/lib a while back and found a similar state - though on the
> >> > pull request builder (apache jenkins). I stopped my java stuff and
> >> > fixed that CI with the help of Aki and Jake reviewing and providing
> >> > guidance. I am not a thrift comitter.
> >> >
> >>
> >> I will say that its discouraging that that CI is now solid red too:
> >> https://builds.apache.org/job/Thrift-precommit/
> >> Part of the answer IMO is for committers to hold a hard line on
> accepting
> >> any patch, or pushing their own, w/o full green CIs.
> >>
> >>
> >> >
> >> >
> >> >>
> >> >>
> >> >> [image: Description: Description: simplivity-lg-xsmall]
> >> >>
> >> >> James E. King, III
> >> >>
> >> >> Architect
> >> >>
> >> >> 8 Technology Drive, 2nd Floor
> >> >> Westborough, MA 01581-1756
> >> >>
> >> >> Ph: 855-SVT-INFO
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ------------------------------
> >> >> PRIVACY STATEMENT:
> >> >> This message is a PRIVATE communication. This message and all
> >> >> attachments are a private communication sent by SimpliVity and are
> >> >> considered to be confidential or protected by privilege. If you are
> >> >> not the intended recipient, you are hereby notified that any
> >> >> disclosure, copying, distribution or use of the information contained
> >> >> in or attached to this message is strictly prohibited. Please notify
> >> >> the sender of the delivery error by replying to this message, and
> then
> >> delete it from your system.
> >> >> For more information please visit http://www.simplivity.com
> >> >> ------------------------------
> >> >>
> >> >
> >> >
> >>
> >>
> ---------------------------------------------------------------------------------------
> >> PRIVACY STATEMENT:
> >> This message is a PRIVATE communication. This message and all
> >> attachments are a private communication sent by SimpliVity and are
> >> considered to be confidential or protected by privilege. If you are not
> the
> >> intended recipient, you are hereby notified that any disclosure,
> copying,
> >> distribution or use of the information contained in or attached to this
> >> message is strictly prohibited. Please notify the sender of the
> delivery
> >> error by replying to this message, and then delete it from your system.
> >>
> >>
> ---------------------------------------------------------------------------------------
> >>
> >
> >
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 8:29 AM, John Sirois <js...@apache.org> wrote:
>
>
> On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com> wrote:
>
>> I got one build through (which failed in "d" tests) and now it's stuck in
>> the same state, see:
>> https://builds.apache.org/job/Thrift-precommit/411/
>>
>> FATAL: Could not checkout master with start point origin/master
>> hudson.plugins.git.GitException: Could not checkout master with start
>> point origin/master
>> at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
>> at
>> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
>> at
>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
>> at
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
>> at
>> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
>> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
>> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
>> at hudson.remoting.Request$2.run(Request.java:326)
>> at
>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> at ......remote call to H10(Native Method)
>> at
>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
>> at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
>> at hudson.remoting.Channel.call(Channel.java:781)
>> at
>> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
>> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
>> at
>> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
>> at
>> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
>> at
>> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
>> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
>> at hudson.scm.SCM.checkout(SCM.java:485)
>> at
>> hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
>> at
>> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
>> at
>> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
>> at
>> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
>> at hudson.model.Run.execute(Run.java:1738)
>> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>> at
>> hudson.model.ResourceController.execute(ResourceController.java:98)
>> at hudson.model.Executor.run(Executor.java:410)
>> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
>> master origin/master" returned status code 1:
>> stdout: lib/lua/TCompactProtocol.lua: needs merge
>>
>> stderr: error: you need to resolve your current index first
>>
>> It looks like the build environment is not forced clean at the beginning
>> of each build.
>>
>
> Ack - looking now.
>
> It is odd that the git portion of these builds went sideways since the
> Jenkins Job Config History auditing plugin shows the last change (before my
> tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its plugins
> were updated by infra causing the previously working job config to not work
> any longer.
>
OK - that analysis was wrong, clearly there has been a change in the build
itself that modifies source code and this causes the issue.
I've enabled <hudson.plugins.git.extensions.impl.CleanBeforeCheckout/> with
the following description:
Clean up the workspace before every checkout by deleting all untracked
files and directories, including those which are specified in .gitignore.
It also resets all *tracked* files to their versioned state. This ensures
that the workspace is in the same state as if you cloned and checked out in
a brand-new empty directory, and ensures that your build is not affected by
the files generated by the previous build.
That sounds like ~ `git clean -fdx && git reset --hard HEAD` to me, which
should do it. That should insulate CI from bad tests that modify checked in
repo state, but those tests shouldn't exist either.
COMMITTERS:
I'd like to reiterate to any committers out there that red CI must be a
hard bright line that is not crossed when accepting patches; otherwise well
be right back here after getting this thing green again. Here we is you -
I won't be interested in helping out a third time if this relapses.
>
>> - Jim
>>
>> -----Original Message-----
>> From: Jim King
>> Sent: Wednesday, April 13, 2016 10:34 PM
>> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
>> Subject: RE: Build Failures
>>
>> The builds were failing claiming that a file was in the middle of being
>> merged and they were all failing early.
>> I think the build environment itself is compromised and there's nothing I
>> can do about that.
>>
>> -----Original Message-----
>> From: John Sirois [mailto:jsirois@apache.org]
>> Sent: Wednesday, April 13, 2016 9:58 PM
>> To: dev@thrift.apache.org
>> Subject: Re: Build Failures
>>
>> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>>
>> >
>> >
>> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
>> wrote:
>> >
>> >> I’m still looking for answers on pull request build failures.
>> >>
>> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> >> precommit builds for strange reasons.
>> >>
>> >> The apache internal builds seem to be failing.
>> >>
>> >
>> > I think the answer is the breaks need a fixer; hopefully you can find
>> > time to help fix.
>> >
>> > I say this because I started down a series of patches to the java
>> > codegen/lib a while back and found a similar state - though on the
>> > pull request builder (apache jenkins). I stopped my java stuff and
>> > fixed that CI with the help of Aki and Jake reviewing and providing
>> > guidance. I am not a thrift comitter.
>> >
>>
>> I will say that its discouraging that that CI is now solid red too:
>> https://builds.apache.org/job/Thrift-precommit/
>> Part of the answer IMO is for committers to hold a hard line on accepting
>> any patch, or pushing their own, w/o full green CIs.
>>
>>
>> >
>> >
>> >>
>> >>
>> >> [image: Description: Description: simplivity-lg-xsmall]
>> >>
>> >> James E. King, III
>> >>
>> >> Architect
>> >>
>> >> 8 Technology Drive, 2nd Floor
>> >> Westborough, MA 01581-1756
>> >>
>> >> Ph: 855-SVT-INFO
>> >>
>> >>
>> >>
>> >>
>> >> ------------------------------
>> >> PRIVACY STATEMENT:
>> >> This message is a PRIVATE communication. This message and all
>> >> attachments are a private communication sent by SimpliVity and are
>> >> considered to be confidential or protected by privilege. If you are
>> >> not the intended recipient, you are hereby notified that any
>> >> disclosure, copying, distribution or use of the information contained
>> >> in or attached to this message is strictly prohibited. Please notify
>> >> the sender of the delivery error by replying to this message, and then
>> delete it from your system.
>> >> For more information please visit http://www.simplivity.com
>> >> ------------------------------
>> >>
>> >
>> >
>>
>> ---------------------------------------------------------------------------------------
>> PRIVACY STATEMENT:
>> This message is a PRIVATE communication. This message and all
>> attachments are a private communication sent by SimpliVity and are
>> considered to be confidential or protected by privilege. If you are not the
>> intended recipient, you are hereby notified that any disclosure, copying,
>> distribution or use of the information contained in or attached to this
>> message is strictly prohibited. Please notify the sender of the delivery
>> error by replying to this message, and then delete it from your system.
>>
>> ---------------------------------------------------------------------------------------
>>
>
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Thu, Apr 14, 2016 at 8:14 AM, Jim King <Ji...@simplivity.com> wrote:
> I got one build through (which failed in "d" tests) and now it's stuck in
> the same state, see:
> https://builds.apache.org/job/Thrift-precommit/411/
>
> FATAL: Could not checkout master with start point origin/master
> hudson.plugins.git.GitException: Could not checkout master with start
> point origin/master
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
> at
> org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
> at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
> at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
> at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
> at hudson.remoting.UserRequest.perform(UserRequest.java:120)
> at hudson.remoting.UserRequest.perform(UserRequest.java:48)
> at hudson.remoting.Request$2.run(Request.java:326)
> at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> at ......remote call to H10(Native Method)
> at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
> at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
> at hudson.remoting.Channel.call(Channel.java:781)
> at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
> at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
> at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
> at
> com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
> at
> hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
> at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
> at hudson.scm.SCM.checkout(SCM.java:485)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
> at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
> at hudson.model.Run.execute(Run.java:1738)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at
> hudson.model.ResourceController.execute(ResourceController.java:98)
> at hudson.model.Executor.run(Executor.java:410)
> Caused by: hudson.plugins.git.GitException: Command "git checkout -b
> master origin/master" returned status code 1:
> stdout: lib/lua/TCompactProtocol.lua: needs merge
>
> stderr: error: you need to resolve your current index first
>
> It looks like the build environment is not forced clean at the beginning
> of each build.
>
Ack - looking now.
It is odd that the git portion of these builds went sideways since the
Jenkins Job Config History auditing plugin shows the last change (before my
tweak last night) was 2016-02-16_02-09-39. I expect jenkins or its plugins
were updated by infra causing the previously working job config to not work
any longer.
> - Jim
>
> -----Original Message-----
> From: Jim King
> Sent: Wednesday, April 13, 2016 10:34 PM
> To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
> Subject: RE: Build Failures
>
> The builds were failing claiming that a file was in the middle of being
> merged and they were all failing early.
> I think the build environment itself is compromised and there's nothing I
> can do about that.
>
> -----Original Message-----
> From: John Sirois [mailto:jsirois@apache.org]
> Sent: Wednesday, April 13, 2016 9:58 PM
> To: dev@thrift.apache.org
> Subject: Re: Build Failures
>
> On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
> >
> >
> > On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com>
> wrote:
> >
> >> I’m still looking for answers on pull request build failures.
> >>
> >> I have 2 or 3 PRs open right now and they’ve failed in the apache
> >> precommit builds for strange reasons.
> >>
> >> The apache internal builds seem to be failing.
> >>
> >
> > I think the answer is the breaks need a fixer; hopefully you can find
> > time to help fix.
> >
> > I say this because I started down a series of patches to the java
> > codegen/lib a while back and found a similar state - though on the
> > pull request builder (apache jenkins). I stopped my java stuff and
> > fixed that CI with the help of Aki and Jake reviewing and providing
> > guidance. I am not a thrift comitter.
> >
>
> I will say that its discouraging that that CI is now solid red too:
> https://builds.apache.org/job/Thrift-precommit/
> Part of the answer IMO is for committers to hold a hard line on accepting
> any patch, or pushing their own, w/o full green CIs.
>
>
> >
> >
> >>
> >>
> >> [image: Description: Description: simplivity-lg-xsmall]
> >>
> >> James E. King, III
> >>
> >> Architect
> >>
> >> 8 Technology Drive, 2nd Floor
> >> Westborough, MA 01581-1756
> >>
> >> Ph: 855-SVT-INFO
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> PRIVACY STATEMENT:
> >> This message is a PRIVATE communication. This message and all
> >> attachments are a private communication sent by SimpliVity and are
> >> considered to be confidential or protected by privilege. If you are
> >> not the intended recipient, you are hereby notified that any
> >> disclosure, copying, distribution or use of the information contained
> >> in or attached to this message is strictly prohibited. Please notify
> >> the sender of the delivery error by replying to this message, and then
> delete it from your system.
> >> For more information please visit http://www.simplivity.com
> >> ------------------------------
> >>
> >
> >
>
> ---------------------------------------------------------------------------------------
> PRIVACY STATEMENT:
> This message is a PRIVATE communication. This message and all attachments
> are a private communication sent by SimpliVity and are considered to be
> confidential or protected by privilege. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of the information contained in or attached to this
> message is strictly prohibited. Please notify the sender of the delivery
> error by replying to this message, and then delete it from your system.
>
> ---------------------------------------------------------------------------------------
>
RE: Build Failures
Posted by Jim King <Ji...@simplivity.com>.
I got one build through (which failed in "d" tests) and now it's stuck in the same state, see:
https://builds.apache.org/job/Thrift-precommit/411/
FATAL: Could not checkout master with start point origin/master
hudson.plugins.git.GitException: Could not checkout master with start point origin/master
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:1962)
at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.checkoutBranch(AbstractGitAPIImpl.java:82)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkoutBranch(CliGitAPIImpl.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to H10(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
at com.sun.proxy.$Proxy115.checkoutBranch(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl.checkoutBranch(RemoteGitImpl.java:327)
at com.cloudbees.jenkins.plugins.git.vmerge.BuildChooserImpl.getCandidateRevisions(BuildChooserImpl.java:78)
at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:951)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1054)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git checkout -b master origin/master" returned status code 1:
stdout: lib/lua/TCompactProtocol.lua: needs merge
stderr: error: you need to resolve your current index first
It looks like the build environment is not forced clean at the beginning of each build.
- Jim
-----Original Message-----
From: Jim King
Sent: Wednesday, April 13, 2016 10:34 PM
To: dev@thrift.apache.org; 'jsirois@apache.org' <js...@apache.org>
Subject: RE: Build Failures
The builds were failing claiming that a file was in the middle of being merged and they were all failing early.
I think the build environment itself is compromised and there's nothing I can do about that.
-----Original Message-----
From: John Sirois [mailto:jsirois@apache.org]
Sent: Wednesday, April 13, 2016 9:58 PM
To: dev@thrift.apache.org
Subject: Re: Build Failures
On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
>
> On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com> wrote:
>
>> I’m still looking for answers on pull request build failures.
>>
>> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> precommit builds for strange reasons.
>>
>> The apache internal builds seem to be failing.
>>
>
> I think the answer is the breaks need a fixer; hopefully you can find
> time to help fix.
>
> I say this because I started down a series of patches to the java
> codegen/lib a while back and found a similar state - though on the
> pull request builder (apache jenkins). I stopped my java stuff and
> fixed that CI with the help of Aki and Jake reviewing and providing
> guidance. I am not a thrift comitter.
>
I will say that its discouraging that that CI is now solid red too:
https://builds.apache.org/job/Thrift-precommit/
Part of the answer IMO is for committers to hold a hard line on accepting any patch, or pushing their own, w/o full green CIs.
>
>
>>
>>
>> [image: Description: Description: simplivity-lg-xsmall]
>>
>> James E. King, III
>>
>> Architect
>>
>> 8 Technology Drive, 2nd Floor
>> Westborough, MA 01581-1756
>>
>> Ph: 855-SVT-INFO
>>
>>
>>
>>
>> ------------------------------
>> PRIVACY STATEMENT:
>> This message is a PRIVATE communication. This message and all
>> attachments are a private communication sent by SimpliVity and are
>> considered to be confidential or protected by privilege. If you are
>> not the intended recipient, you are hereby notified that any
>> disclosure, copying, distribution or use of the information contained
>> in or attached to this message is strictly prohibited. Please notify
>> the sender of the delivery error by replying to this message, and then delete it from your system.
>> For more information please visit http://www.simplivity.com
>> ------------------------------
>>
>
>
---------------------------------------------------------------------------------------
PRIVACY STATEMENT:
This message is a PRIVATE communication. This message and all attachments are a private communication sent by SimpliVity and are considered to be confidential or protected by privilege. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the information contained in or attached to this message is strictly prohibited. Please notify the sender of the delivery error by replying to this message, and then delete it from your system.
---------------------------------------------------------------------------------------
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Wed, Apr 13, 2016 at 7:54 PM, John Sirois <js...@apache.org> wrote:
>
>
> On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com> wrote:
>
>> I’m still looking for answers on pull request build failures.
>>
>> I have 2 or 3 PRs open right now and they’ve failed in the apache
>> precommit builds for strange reasons.
>>
>> The apache internal builds seem to be failing.
>>
>
> I think the answer is the breaks need a fixer; hopefully you can find time
> to help fix.
>
> I say this because I started down a series of patches to the java
> codegen/lib a while back and found a similar state - though on the pull
> request builder (apache jenkins). I stopped my java stuff and fixed that
> CI with the help of Aki and Jake reviewing and providing guidance. I am
> not a thrift comitter.
>
I will say that its discouraging that that CI is now solid red too:
https://builds.apache.org/job/Thrift-precommit/
Part of the answer IMO is for committers to hold a hard line on accepting
any patch, or pushing their own, w/o full green CIs.
>
>
>>
>>
>> [image: Description: Description: simplivity-lg-xsmall]
>>
>> James E. King, III
>>
>> Architect
>>
>> 8 Technology Drive, 2nd Floor
>> Westborough, MA 01581-1756
>>
>> Ph: 855-SVT-INFO
>>
>>
>>
>>
>> ------------------------------
>> PRIVACY STATEMENT:
>> This message is a PRIVATE communication. This message and all attachments
>> are a private communication sent by SimpliVity and are considered to be
>> confidential or protected by privilege. If you are not the intended
>> recipient, you are hereby notified that any disclosure, copying,
>> distribution or use of the information contained in or attached to this
>> message is strictly prohibited. Please notify the sender of the delivery
>> error by replying to this message, and then delete it from your system.
>> For more information please visit http://www.simplivity.com
>> ------------------------------
>>
>
>
Re: Build Failures
Posted by John Sirois <js...@apache.org>.
On Wed, Apr 13, 2016 at 7:51 PM, Jim King <Ji...@simplivity.com> wrote:
> I’m still looking for answers on pull request build failures.
>
> I have 2 or 3 PRs open right now and they’ve failed in the apache
> precommit builds for strange reasons.
>
> The apache internal builds seem to be failing.
>
I think the answer is the breaks need a fixer; hopefully you can find time
to help fix.
I say this because I started down a series of patches to the java
codegen/lib a while back and found a similar state - though on the pull
request builder (apache jenkins). I stopped my java stuff and fixed that
CI with the help of Aki and Jake reviewing and providing guidance. I am
not a thrift comitter.
>
>
> [image: Description: Description: simplivity-lg-xsmall]
>
> James E. King, III
>
> Architect
>
> 8 Technology Drive, 2nd Floor
> Westborough, MA 01581-1756
>
> Ph: 855-SVT-INFO
>
>
>
>
> ------------------------------
> PRIVACY STATEMENT:
> This message is a PRIVATE communication. This message and all attachments
> are a private communication sent by SimpliVity and are considered to be
> confidential or protected by privilege. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of the information contained in or attached to this
> message is strictly prohibited. Please notify the sender of the delivery
> error by replying to this message, and then delete it from your system.
> For more information please visit http://www.simplivity.com
> ------------------------------
>