You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Kihwal Lee <ki...@yahoo-inc.com.INVALID> on 2015/02/05 18:49:31 UTC
Erratic Jenkins behavior
I am sure many of us have seen strange jenkins behavior out of the precommit builds.
- build artifacts missing
- serving build artifact belonging to another build. This also causes wrong precommit results to be posted on the bug.
- etc.
The latest one I saw is disappearance of the unit test stdout/stderr file during a build. After a successful run of unit tests, the file vanished, so the script could not cat it. It looked like another build process had deleted it, while this build was in progress.
It might have something to do with the fact that the patch-dir is set like following:
PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI don't have access to the jenkins build configs or the build machines, so I can't debug it further, but I think we need to take care of it sooner than later. Can any one help?
Kihwal
Re: Erratic Jenkins behavior
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Sorry I don¹t have much insight into this either. Our Jenkins setup is a
bit of a black box to me.
Too few of us have access to the build hosts and it makes debugging
difficult. Perhaps we should grant access to all committers.
On 2/9/15, 11:29 AM, "Colin P. McCabe" <cm...@apache.org> wrote:
>I'm sorry, I don't have any insight into this. With regard to
>HADOOP-11084, I thought that $BUILD_URL would be unique for each
>concurrent build, which would prevent build artifacts from getting
>mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>posted, perhaps this is not the case? Perhaps someone can explain how
>this is supposed to work (I am a Jenkins newbie).
>
>regards,
>Colin
>
>On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>><ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>file
>>> during a build. After a successful run of unit tests, the file
>>>vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set
>>>like
>>> following:
>>>
>>>
>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/
>>>../patchprocessI
>>> don't have access to the jenkins build configs or the build machines,
>>>so I
>>> can't debug it further, but I think we need to take care of it sooner
>>>than
>>> later. Can any one help?
>>>
>>> Kihwal
>>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
Hmm. I guess my thought would be that we would have a fixed number of
"slots" (i.e. executors on a single node with associated .m2
directories). Then we wouldn't clear each .m2 in between runs, but we
would ensure that only one slot at a time had access to each
directory.
In that case, build times wouldn't increase that much (or really at
all, until a dependency changed... right?). When a dependency changed
we'd have to do O(N_slots) amount of work, but dependencies don't
change that often.
Of course, the current situation also generates a lot of extra work
because people need to rekick builds that failed for mystery reasons.
cheers.
Colin
On Wed, Feb 18, 2015 at 9:53 AM, Chris Nauroth <cn...@hortonworks.com> wrote:
> I¹m pretty sure there is no guarantee of isolation on a shared
> .m2/repository directory for multiple concurrent Maven processes. I¹ve
> had a theory for a while that one build running ³mvm install² can
> overwrite the snapshot artifact that was just installed by another
> concurrent build. This can create bizarre problems, for example if a
> patch introduces a new class in hadoop-common and then references that
> class from hadoop-hdfs.
>
> I expect using completely separate work directories for .m2/repository,
> the patch directory, and the Jenkins workspace could resolve this. The
> typical cost for this kind of change is increased disk consumption and
> increased build time, since Maven would need to download dependencies
> fresh every time.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 2/12/15, 2:00 PM, "Colin P. McCabe" <cm...@apache.org> wrote:
>
>>We could potentially use different .m2 directories for each executor.
>>I think this has been brought up in the past as well.
>>
>>I'm not sure how maven handles concurrent access to the .m2
>>directory... if it's not using flock or fnctl then it's not really
>>safe. This might explain some of our missing class error issues.
>>
>>Colin
>>
>>On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com>
>>wrote:
>>> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things
>>>from other builds if they ended up published to ~/.m2/repository during
>>>the process
>>>
>>>
>>>
>>> On 9 February 2015 at 19:29:06, Colin P. McCabe
>>>(cmccabe@apache.org<ma...@apache.org>) wrote:
>>>
>>> I'm sorry, I don't have any insight into this. With regard to
>>> HADOOP-11084, I thought that $BUILD_URL would be unique for each
>>> concurrent build, which would prevent build artifacts from getting
>>> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>>> posted, perhaps this is not the case? Perhaps someone can explain how
>>> this is supposed to work (I am a Jenkins newbie).
>>>
>>> regards,
>>> Colin
>>>
>>> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>>>wrote:
>>>> Thanks Kihwal for bringing this up.
>>>>
>>>> Seems related to:
>>>>
>>>> https://issues.apache.org/jira/browse/HADOOP-11084
>>>>
>>>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>>>> insight about the issue Kihwal described?
>>>>
>>>> Thanks.
>>>>
>>>> --Yongjun
>>>>
>>>>
>>>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>>><ki...@yahoo-inc.com.invalid>
>>>> wrote:
>>>>
>>>>> I am sure many of us have seen strange jenkins behavior out of the
>>>>> precommit builds.
>>>>>
>>>>> - build artifacts missing
>>>>> - serving build artifact belonging to another build. This also causes
>>>>> wrong precommit results to be posted on the bug.
>>>>> - etc.
>>>>>
>>>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>>>file
>>>>> during a build. After a successful run of unit tests, the file
>>>>>vanished, so
>>>>> the script could not cat it. It looked like another build process had
>>>>> deleted it, while this build was in progress.
>>>>>
>>>>> It might have something to do with the fact that the patch-dir is set
>>>>>like
>>>>> following:
>>>>>
>>>>>
>>>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build
>>>>>/../patchprocessI
>>>>> don't have access to the jenkins build configs or the build machines,
>>>>>so I
>>>>> can't debug it further, but I think we need to take care of it sooner
>>>>>than
>>>>> later. Can any one help?
>>>>>
>>>>> Kihwal
>>>>>
>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
Hmm. I guess my thought would be that we would have a fixed number of
"slots" (i.e. executors on a single node with associated .m2
directories). Then we wouldn't clear each .m2 in between runs, but we
would ensure that only one slot at a time had access to each
directory.
In that case, build times wouldn't increase that much (or really at
all, until a dependency changed... right?). When a dependency changed
we'd have to do O(N_slots) amount of work, but dependencies don't
change that often.
Of course, the current situation also generates a lot of extra work
because people need to rekick builds that failed for mystery reasons.
cheers.
Colin
On Wed, Feb 18, 2015 at 9:53 AM, Chris Nauroth <cn...@hortonworks.com> wrote:
> I¹m pretty sure there is no guarantee of isolation on a shared
> .m2/repository directory for multiple concurrent Maven processes. I¹ve
> had a theory for a while that one build running ³mvm install² can
> overwrite the snapshot artifact that was just installed by another
> concurrent build. This can create bizarre problems, for example if a
> patch introduces a new class in hadoop-common and then references that
> class from hadoop-hdfs.
>
> I expect using completely separate work directories for .m2/repository,
> the patch directory, and the Jenkins workspace could resolve this. The
> typical cost for this kind of change is increased disk consumption and
> increased build time, since Maven would need to download dependencies
> fresh every time.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 2/12/15, 2:00 PM, "Colin P. McCabe" <cm...@apache.org> wrote:
>
>>We could potentially use different .m2 directories for each executor.
>>I think this has been brought up in the past as well.
>>
>>I'm not sure how maven handles concurrent access to the .m2
>>directory... if it's not using flock or fnctl then it's not really
>>safe. This might explain some of our missing class error issues.
>>
>>Colin
>>
>>On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com>
>>wrote:
>>> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things
>>>from other builds if they ended up published to ~/.m2/repository during
>>>the process
>>>
>>>
>>>
>>> On 9 February 2015 at 19:29:06, Colin P. McCabe
>>>(cmccabe@apache.org<ma...@apache.org>) wrote:
>>>
>>> I'm sorry, I don't have any insight into this. With regard to
>>> HADOOP-11084, I thought that $BUILD_URL would be unique for each
>>> concurrent build, which would prevent build artifacts from getting
>>> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>>> posted, perhaps this is not the case? Perhaps someone can explain how
>>> this is supposed to work (I am a Jenkins newbie).
>>>
>>> regards,
>>> Colin
>>>
>>> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>>>wrote:
>>>> Thanks Kihwal for bringing this up.
>>>>
>>>> Seems related to:
>>>>
>>>> https://issues.apache.org/jira/browse/HADOOP-11084
>>>>
>>>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>>>> insight about the issue Kihwal described?
>>>>
>>>> Thanks.
>>>>
>>>> --Yongjun
>>>>
>>>>
>>>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>>><ki...@yahoo-inc.com.invalid>
>>>> wrote:
>>>>
>>>>> I am sure many of us have seen strange jenkins behavior out of the
>>>>> precommit builds.
>>>>>
>>>>> - build artifacts missing
>>>>> - serving build artifact belonging to another build. This also causes
>>>>> wrong precommit results to be posted on the bug.
>>>>> - etc.
>>>>>
>>>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>>>file
>>>>> during a build. After a successful run of unit tests, the file
>>>>>vanished, so
>>>>> the script could not cat it. It looked like another build process had
>>>>> deleted it, while this build was in progress.
>>>>>
>>>>> It might have something to do with the fact that the patch-dir is set
>>>>>like
>>>>> following:
>>>>>
>>>>>
>>>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build
>>>>>/../patchprocessI
>>>>> don't have access to the jenkins build configs or the build machines,
>>>>>so I
>>>>> can't debug it further, but I think we need to take care of it sooner
>>>>>than
>>>>> later. Can any one help?
>>>>>
>>>>> Kihwal
>>>>>
>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
Hmm. I guess my thought would be that we would have a fixed number of
"slots" (i.e. executors on a single node with associated .m2
directories). Then we wouldn't clear each .m2 in between runs, but we
would ensure that only one slot at a time had access to each
directory.
In that case, build times wouldn't increase that much (or really at
all, until a dependency changed... right?). When a dependency changed
we'd have to do O(N_slots) amount of work, but dependencies don't
change that often.
Of course, the current situation also generates a lot of extra work
because people need to rekick builds that failed for mystery reasons.
cheers.
Colin
On Wed, Feb 18, 2015 at 9:53 AM, Chris Nauroth <cn...@hortonworks.com> wrote:
> I¹m pretty sure there is no guarantee of isolation on a shared
> .m2/repository directory for multiple concurrent Maven processes. I¹ve
> had a theory for a while that one build running ³mvm install² can
> overwrite the snapshot artifact that was just installed by another
> concurrent build. This can create bizarre problems, for example if a
> patch introduces a new class in hadoop-common and then references that
> class from hadoop-hdfs.
>
> I expect using completely separate work directories for .m2/repository,
> the patch directory, and the Jenkins workspace could resolve this. The
> typical cost for this kind of change is increased disk consumption and
> increased build time, since Maven would need to download dependencies
> fresh every time.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
>
>
>
> On 2/12/15, 2:00 PM, "Colin P. McCabe" <cm...@apache.org> wrote:
>
>>We could potentially use different .m2 directories for each executor.
>>I think this has been brought up in the past as well.
>>
>>I'm not sure how maven handles concurrent access to the .m2
>>directory... if it's not using flock or fnctl then it's not really
>>safe. This might explain some of our missing class error issues.
>>
>>Colin
>>
>>On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com>
>>wrote:
>>> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things
>>>from other builds if they ended up published to ~/.m2/repository during
>>>the process
>>>
>>>
>>>
>>> On 9 February 2015 at 19:29:06, Colin P. McCabe
>>>(cmccabe@apache.org<ma...@apache.org>) wrote:
>>>
>>> I'm sorry, I don't have any insight into this. With regard to
>>> HADOOP-11084, I thought that $BUILD_URL would be unique for each
>>> concurrent build, which would prevent build artifacts from getting
>>> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>>> posted, perhaps this is not the case? Perhaps someone can explain how
>>> this is supposed to work (I am a Jenkins newbie).
>>>
>>> regards,
>>> Colin
>>>
>>> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>>>wrote:
>>>> Thanks Kihwal for bringing this up.
>>>>
>>>> Seems related to:
>>>>
>>>> https://issues.apache.org/jira/browse/HADOOP-11084
>>>>
>>>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>>>> insight about the issue Kihwal described?
>>>>
>>>> Thanks.
>>>>
>>>> --Yongjun
>>>>
>>>>
>>>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>>><ki...@yahoo-inc.com.invalid>
>>>> wrote:
>>>>
>>>>> I am sure many of us have seen strange jenkins behavior out of the
>>>>> precommit builds.
>>>>>
>>>>> - build artifacts missing
>>>>> - serving build artifact belonging to another build. This also causes
>>>>> wrong precommit results to be posted on the bug.
>>>>> - etc.
>>>>>
>>>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>>>file
>>>>> during a build. After a successful run of unit tests, the file
>>>>>vanished, so
>>>>> the script could not cat it. It looked like another build process had
>>>>> deleted it, while this build was in progress.
>>>>>
>>>>> It might have something to do with the fact that the patch-dir is set
>>>>>like
>>>>> following:
>>>>>
>>>>>
>>>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build
>>>>>/../patchprocessI
>>>>> don't have access to the jenkins build configs or the build machines,
>>>>>so I
>>>>> can't debug it further, but I think we need to take care of it sooner
>>>>>than
>>>>> later. Can any one help?
>>>>>
>>>>> Kihwal
>>>>>
>
Re: Erratic Jenkins behavior
Posted by Chris Nauroth <cn...@hortonworks.com>.
I¹m pretty sure there is no guarantee of isolation on a shared
.m2/repository directory for multiple concurrent Maven processes. I¹ve
had a theory for a while that one build running ³mvm install² can
overwrite the snapshot artifact that was just installed by another
concurrent build. This can create bizarre problems, for example if a
patch introduces a new class in hadoop-common and then references that
class from hadoop-hdfs.
I expect using completely separate work directories for .m2/repository,
the patch directory, and the Jenkins workspace could resolve this. The
typical cost for this kind of change is increased disk consumption and
increased build time, since Maven would need to download dependencies
fresh every time.
Chris Nauroth
Hortonworks
http://hortonworks.com/
On 2/12/15, 2:00 PM, "Colin P. McCabe" <cm...@apache.org> wrote:
>We could potentially use different .m2 directories for each executor.
>I think this has been brought up in the past as well.
>
>I'm not sure how maven handles concurrent access to the .m2
>directory... if it's not using flock or fnctl then it's not really
>safe. This might explain some of our missing class error issues.
>
>Colin
>
>On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com>
>wrote:
>> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things
>>from other builds if they ended up published to ~/.m2/repository during
>>the process
>>
>>
>>
>> On 9 February 2015 at 19:29:06, Colin P. McCabe
>>(cmccabe@apache.org<ma...@apache.org>) wrote:
>>
>> I'm sorry, I don't have any insight into this. With regard to
>> HADOOP-11084, I thought that $BUILD_URL would be unique for each
>> concurrent build, which would prevent build artifacts from getting
>> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>> posted, perhaps this is not the case? Perhaps someone can explain how
>> this is supposed to work (I am a Jenkins newbie).
>>
>> regards,
>> Colin
>>
>> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>>wrote:
>>> Thanks Kihwal for bringing this up.
>>>
>>> Seems related to:
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-11084
>>>
>>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>>> insight about the issue Kihwal described?
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>><ki...@yahoo-inc.com.invalid>
>>> wrote:
>>>
>>>> I am sure many of us have seen strange jenkins behavior out of the
>>>> precommit builds.
>>>>
>>>> - build artifacts missing
>>>> - serving build artifact belonging to another build. This also causes
>>>> wrong precommit results to be posted on the bug.
>>>> - etc.
>>>>
>>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>>file
>>>> during a build. After a successful run of unit tests, the file
>>>>vanished, so
>>>> the script could not cat it. It looked like another build process had
>>>> deleted it, while this build was in progress.
>>>>
>>>> It might have something to do with the fact that the patch-dir is set
>>>>like
>>>> following:
>>>>
>>>>
>>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build
>>>>/../patchprocessI
>>>> don't have access to the jenkins build configs or the build machines,
>>>>so I
>>>> can't debug it further, but I think we need to take care of it sooner
>>>>than
>>>> later. Can any one help?
>>>>
>>>> Kihwal
>>>>
Re: Erratic Jenkins behavior
Posted by Chris Nauroth <cn...@hortonworks.com>.
I¹m pretty sure there is no guarantee of isolation on a shared
.m2/repository directory for multiple concurrent Maven processes. I¹ve
had a theory for a while that one build running ³mvm install² can
overwrite the snapshot artifact that was just installed by another
concurrent build. This can create bizarre problems, for example if a
patch introduces a new class in hadoop-common and then references that
class from hadoop-hdfs.
I expect using completely separate work directories for .m2/repository,
the patch directory, and the Jenkins workspace could resolve this. The
typical cost for this kind of change is increased disk consumption and
increased build time, since Maven would need to download dependencies
fresh every time.
Chris Nauroth
Hortonworks
http://hortonworks.com/
On 2/12/15, 2:00 PM, "Colin P. McCabe" <cm...@apache.org> wrote:
>We could potentially use different .m2 directories for each executor.
>I think this has been brought up in the past as well.
>
>I'm not sure how maven handles concurrent access to the .m2
>directory... if it's not using flock or fnctl then it's not really
>safe. This might explain some of our missing class error issues.
>
>Colin
>
>On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com>
>wrote:
>> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things
>>from other builds if they ended up published to ~/.m2/repository during
>>the process
>>
>>
>>
>> On 9 February 2015 at 19:29:06, Colin P. McCabe
>>(cmccabe@apache.org<ma...@apache.org>) wrote:
>>
>> I'm sorry, I don't have any insight into this. With regard to
>> HADOOP-11084, I thought that $BUILD_URL would be unique for each
>> concurrent build, which would prevent build artifacts from getting
>> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>> posted, perhaps this is not the case? Perhaps someone can explain how
>> this is supposed to work (I am a Jenkins newbie).
>>
>> regards,
>> Colin
>>
>> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>>wrote:
>>> Thanks Kihwal for bringing this up.
>>>
>>> Seems related to:
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-11084
>>>
>>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>>> insight about the issue Kihwal described?
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>><ki...@yahoo-inc.com.invalid>
>>> wrote:
>>>
>>>> I am sure many of us have seen strange jenkins behavior out of the
>>>> precommit builds.
>>>>
>>>> - build artifacts missing
>>>> - serving build artifact belonging to another build. This also causes
>>>> wrong precommit results to be posted on the bug.
>>>> - etc.
>>>>
>>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>>file
>>>> during a build. After a successful run of unit tests, the file
>>>>vanished, so
>>>> the script could not cat it. It looked like another build process had
>>>> deleted it, while this build was in progress.
>>>>
>>>> It might have something to do with the fact that the patch-dir is set
>>>>like
>>>> following:
>>>>
>>>>
>>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build
>>>>/../patchprocessI
>>>> don't have access to the jenkins build configs or the build machines,
>>>>so I
>>>> can't debug it further, but I think we need to take care of it sooner
>>>>than
>>>> later. Can any one help?
>>>>
>>>> Kihwal
>>>>
Re: Erratic Jenkins behavior
Posted by Chris Nauroth <cn...@hortonworks.com>.
I¹m pretty sure there is no guarantee of isolation on a shared
.m2/repository directory for multiple concurrent Maven processes. I¹ve
had a theory for a while that one build running ³mvm install² can
overwrite the snapshot artifact that was just installed by another
concurrent build. This can create bizarre problems, for example if a
patch introduces a new class in hadoop-common and then references that
class from hadoop-hdfs.
I expect using completely separate work directories for .m2/repository,
the patch directory, and the Jenkins workspace could resolve this. The
typical cost for this kind of change is increased disk consumption and
increased build time, since Maven would need to download dependencies
fresh every time.
Chris Nauroth
Hortonworks
http://hortonworks.com/
On 2/12/15, 2:00 PM, "Colin P. McCabe" <cm...@apache.org> wrote:
>We could potentially use different .m2 directories for each executor.
>I think this has been brought up in the past as well.
>
>I'm not sure how maven handles concurrent access to the .m2
>directory... if it's not using flock or fnctl then it's not really
>safe. This might explain some of our missing class error issues.
>
>Colin
>
>On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com>
>wrote:
>> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things
>>from other builds if they ended up published to ~/.m2/repository during
>>the process
>>
>>
>>
>> On 9 February 2015 at 19:29:06, Colin P. McCabe
>>(cmccabe@apache.org<ma...@apache.org>) wrote:
>>
>> I'm sorry, I don't have any insight into this. With regard to
>> HADOOP-11084, I thought that $BUILD_URL would be unique for each
>> concurrent build, which would prevent build artifacts from getting
>> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>> posted, perhaps this is not the case? Perhaps someone can explain how
>> this is supposed to work (I am a Jenkins newbie).
>>
>> regards,
>> Colin
>>
>> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>>wrote:
>>> Thanks Kihwal for bringing this up.
>>>
>>> Seems related to:
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-11084
>>>
>>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>>> insight about the issue Kihwal described?
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>><ki...@yahoo-inc.com.invalid>
>>> wrote:
>>>
>>>> I am sure many of us have seen strange jenkins behavior out of the
>>>> precommit builds.
>>>>
>>>> - build artifacts missing
>>>> - serving build artifact belonging to another build. This also causes
>>>> wrong precommit results to be posted on the bug.
>>>> - etc.
>>>>
>>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>>file
>>>> during a build. After a successful run of unit tests, the file
>>>>vanished, so
>>>> the script could not cat it. It looked like another build process had
>>>> deleted it, while this build was in progress.
>>>>
>>>> It might have something to do with the fact that the patch-dir is set
>>>>like
>>>> following:
>>>>
>>>>
>>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build
>>>>/../patchprocessI
>>>> don't have access to the jenkins build configs or the build machines,
>>>>so I
>>>> can't debug it further, but I think we need to take care of it sooner
>>>>than
>>>> later. Can any one help?
>>>>
>>>> Kihwal
>>>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
We could potentially use different .m2 directories for each executor.
I think this has been brought up in the past as well.
I'm not sure how maven handles concurrent access to the .m2
directory... if it's not using flock or fnctl then it's not really
safe. This might explain some of our missing class error issues.
Colin
On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com> wrote:
> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things from other builds if they ended up published to ~/.m2/repository during the process
>
>
>
> On 9 February 2015 at 19:29:06, Colin P. McCabe (cmccabe@apache.org<ma...@apache.org>) wrote:
>
> I'm sorry, I don't have any insight into this. With regard to
> HADOOP-11084, I thought that $BUILD_URL would be unique for each
> concurrent build, which would prevent build artifacts from getting
> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
> posted, perhaps this is not the case? Perhaps someone can explain how
> this is supposed to work (I am a Jenkins newbie).
>
> regards,
> Colin
>
> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr file
>>> during a build. After a successful run of unit tests, the file vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set like
>>> following:
>>>
>>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>>> don't have access to the jenkins build configs or the build machines, so I
>>> can't debug it further, but I think we need to take care of it sooner than
>>> later. Can any one help?
>>>
>>> Kihwal
>>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
We could potentially use different .m2 directories for each executor.
I think this has been brought up in the past as well.
I'm not sure how maven handles concurrent access to the .m2
directory... if it's not using flock or fnctl then it's not really
safe. This might explain some of our missing class error issues.
Colin
On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com> wrote:
> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things from other builds if they ended up published to ~/.m2/repository during the process
>
>
>
> On 9 February 2015 at 19:29:06, Colin P. McCabe (cmccabe@apache.org<ma...@apache.org>) wrote:
>
> I'm sorry, I don't have any insight into this. With regard to
> HADOOP-11084, I thought that $BUILD_URL would be unique for each
> concurrent build, which would prevent build artifacts from getting
> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
> posted, perhaps this is not the case? Perhaps someone can explain how
> this is supposed to work (I am a Jenkins newbie).
>
> regards,
> Colin
>
> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr file
>>> during a build. After a successful run of unit tests, the file vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set like
>>> following:
>>>
>>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>>> don't have access to the jenkins build configs or the build machines, so I
>>> can't debug it further, but I think we need to take care of it sooner than
>>> later. Can any one help?
>>>
>>> Kihwal
>>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
We could potentially use different .m2 directories for each executor.
I think this has been brought up in the past as well.
I'm not sure how maven handles concurrent access to the .m2
directory... if it's not using flock or fnctl then it's not really
safe. This might explain some of our missing class error issues.
Colin
On Tue, Feb 10, 2015 at 2:13 AM, Steve Loughran <st...@hortonworks.com> wrote:
> Mvn is a dark mystery to us all. I wouldn't trust it not pick up things from other builds if they ended up published to ~/.m2/repository during the process
>
>
>
> On 9 February 2015 at 19:29:06, Colin P. McCabe (cmccabe@apache.org<ma...@apache.org>) wrote:
>
> I'm sorry, I don't have any insight into this. With regard to
> HADOOP-11084, I thought that $BUILD_URL would be unique for each
> concurrent build, which would prevent build artifacts from getting
> mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
> posted, perhaps this is not the case? Perhaps someone can explain how
> this is supposed to work (I am a Jenkins newbie).
>
> regards,
> Colin
>
> On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr file
>>> during a build. After a successful run of unit tests, the file vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set like
>>> following:
>>>
>>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>>> don't have access to the jenkins build configs or the build machines, so I
>>> can't debug it further, but I think we need to take care of it sooner than
>>> later. Can any one help?
>>>
>>> Kihwal
>>>
Re: Erratic Jenkins behavior
Posted by Steve Loughran <st...@hortonworks.com>.
Mvn is a dark mystery to us all. I wouldn't trust it not pick up things from other builds if they ended up published to ~/.m2/repository during the process
On 9 February 2015 at 19:29:06, Colin P. McCabe (cmccabe@apache.org<ma...@apache.org>) wrote:
I'm sorry, I don't have any insight into this. With regard to
HADOOP-11084, I thought that $BUILD_URL would be unique for each
concurrent build, which would prevent build artifacts from getting
mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
posted, perhaps this is not the case? Perhaps someone can explain how
this is supposed to work (I am a Jenkins newbie).
regards,
Colin
On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
> Thanks Kihwal for bringing this up.
>
> Seems related to:
>
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
> insight about the issue Kihwal described?
>
> Thanks.
>
> --Yongjun
>
>
> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
>> I am sure many of us have seen strange jenkins behavior out of the
>> precommit builds.
>>
>> - build artifacts missing
>> - serving build artifact belonging to another build. This also causes
>> wrong precommit results to be posted on the bug.
>> - etc.
>>
>> The latest one I saw is disappearance of the unit test stdout/stderr file
>> during a build. After a successful run of unit tests, the file vanished, so
>> the script could not cat it. It looked like another build process had
>> deleted it, while this build was in progress.
>>
>> It might have something to do with the fact that the patch-dir is set like
>> following:
>>
>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>> don't have access to the jenkins build configs or the build machines, so I
>> can't debug it further, but I think we need to take care of it sooner than
>> later. Can any one help?
>>
>> Kihwal
>>
Re: Erratic Jenkins behavior
Posted by Steve Loughran <st...@hortonworks.com>.
Mvn is a dark mystery to us all. I wouldn't trust it not pick up things from other builds if they ended up published to ~/.m2/repository during the process
On 9 February 2015 at 19:29:06, Colin P. McCabe (cmccabe@apache.org<ma...@apache.org>) wrote:
I'm sorry, I don't have any insight into this. With regard to
HADOOP-11084, I thought that $BUILD_URL would be unique for each
concurrent build, which would prevent build artifacts from getting
mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
posted, perhaps this is not the case? Perhaps someone can explain how
this is supposed to work (I am a Jenkins newbie).
regards,
Colin
On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
> Thanks Kihwal for bringing this up.
>
> Seems related to:
>
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
> insight about the issue Kihwal described?
>
> Thanks.
>
> --Yongjun
>
>
> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
>> I am sure many of us have seen strange jenkins behavior out of the
>> precommit builds.
>>
>> - build artifacts missing
>> - serving build artifact belonging to another build. This also causes
>> wrong precommit results to be posted on the bug.
>> - etc.
>>
>> The latest one I saw is disappearance of the unit test stdout/stderr file
>> during a build. After a successful run of unit tests, the file vanished, so
>> the script could not cat it. It looked like another build process had
>> deleted it, while this build was in progress.
>>
>> It might have something to do with the fact that the patch-dir is set like
>> following:
>>
>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>> don't have access to the jenkins build configs or the build machines, so I
>> can't debug it further, but I think we need to take care of it sooner than
>> later. Can any one help?
>>
>> Kihwal
>>
Re: Erratic Jenkins behavior
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Sorry I don¹t have much insight into this either. Our Jenkins setup is a
bit of a black box to me.
Too few of us have access to the build hosts and it makes debugging
difficult. Perhaps we should grant access to all committers.
On 2/9/15, 11:29 AM, "Colin P. McCabe" <cm...@apache.org> wrote:
>I'm sorry, I don't have any insight into this. With regard to
>HADOOP-11084, I thought that $BUILD_URL would be unique for each
>concurrent build, which would prevent build artifacts from getting
>mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>posted, perhaps this is not the case? Perhaps someone can explain how
>this is supposed to work (I am a Jenkins newbie).
>
>regards,
>Colin
>
>On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>><ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>file
>>> during a build. After a successful run of unit tests, the file
>>>vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set
>>>like
>>> following:
>>>
>>>
>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/
>>>../patchprocessI
>>> don't have access to the jenkins build configs or the build machines,
>>>so I
>>> can't debug it further, but I think we need to take care of it sooner
>>>than
>>> later. Can any one help?
>>>
>>> Kihwal
>>>
Re: Erratic Jenkins behavior
Posted by Steve Loughran <st...@hortonworks.com>.
Mvn is a dark mystery to us all. I wouldn't trust it not pick up things from other builds if they ended up published to ~/.m2/repository during the process
On 9 February 2015 at 19:29:06, Colin P. McCabe (cmccabe@apache.org<ma...@apache.org>) wrote:
I'm sorry, I don't have any insight into this. With regard to
HADOOP-11084, I thought that $BUILD_URL would be unique for each
concurrent build, which would prevent build artifacts from getting
mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
posted, perhaps this is not the case? Perhaps someone can explain how
this is supposed to work (I am a Jenkins newbie).
regards,
Colin
On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
> Thanks Kihwal for bringing this up.
>
> Seems related to:
>
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
> insight about the issue Kihwal described?
>
> Thanks.
>
> --Yongjun
>
>
> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
>> I am sure many of us have seen strange jenkins behavior out of the
>> precommit builds.
>>
>> - build artifacts missing
>> - serving build artifact belonging to another build. This also causes
>> wrong precommit results to be posted on the bug.
>> - etc.
>>
>> The latest one I saw is disappearance of the unit test stdout/stderr file
>> during a build. After a successful run of unit tests, the file vanished, so
>> the script could not cat it. It looked like another build process had
>> deleted it, while this build was in progress.
>>
>> It might have something to do with the fact that the patch-dir is set like
>> following:
>>
>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>> don't have access to the jenkins build configs or the build machines, so I
>> can't debug it further, but I think we need to take care of it sooner than
>> later. Can any one help?
>>
>> Kihwal
>>
Re: Erratic Jenkins behavior
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Sorry I don¹t have much insight into this either. Our Jenkins setup is a
bit of a black box to me.
Too few of us have access to the build hosts and it makes debugging
difficult. Perhaps we should grant access to all committers.
On 2/9/15, 11:29 AM, "Colin P. McCabe" <cm...@apache.org> wrote:
>I'm sorry, I don't have any insight into this. With regard to
>HADOOP-11084, I thought that $BUILD_URL would be unique for each
>concurrent build, which would prevent build artifacts from getting
>mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
>posted, perhaps this is not the case? Perhaps someone can explain how
>this is supposed to work (I am a Jenkins newbie).
>
>regards,
>Colin
>
>On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com>
>wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>><ki...@yahoo-inc.com.invalid>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>file
>>> during a build. After a successful run of unit tests, the file
>>>vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set
>>>like
>>> following:
>>>
>>>
>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/
>>>../patchprocessI
>>> don't have access to the jenkins build configs or the build machines,
>>>so I
>>> can't debug it further, but I think we need to take care of it sooner
>>>than
>>> later. Can any one help?
>>>
>>> Kihwal
>>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
I'm sorry, I don't have any insight into this. With regard to
HADOOP-11084, I thought that $BUILD_URL would be unique for each
concurrent build, which would prevent build artifacts from getting
mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
posted, perhaps this is not the case? Perhaps someone can explain how
this is supposed to work (I am a Jenkins newbie).
regards,
Colin
On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
> Thanks Kihwal for bringing this up.
>
> Seems related to:
>
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
> insight about the issue Kihwal described?
>
> Thanks.
>
> --Yongjun
>
>
> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
>> I am sure many of us have seen strange jenkins behavior out of the
>> precommit builds.
>>
>> - build artifacts missing
>> - serving build artifact belonging to another build. This also causes
>> wrong precommit results to be posted on the bug.
>> - etc.
>>
>> The latest one I saw is disappearance of the unit test stdout/stderr file
>> during a build. After a successful run of unit tests, the file vanished, so
>> the script could not cat it. It looked like another build process had
>> deleted it, while this build was in progress.
>>
>> It might have something to do with the fact that the patch-dir is set like
>> following:
>>
>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>> don't have access to the jenkins build configs or the build machines, so I
>> can't debug it further, but I think we need to take care of it sooner than
>> later. Can any one help?
>>
>> Kihwal
>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
I'm sorry, I don't have any insight into this. With regard to
HADOOP-11084, I thought that $BUILD_URL would be unique for each
concurrent build, which would prevent build artifacts from getting
mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
posted, perhaps this is not the case? Perhaps someone can explain how
this is supposed to work (I am a Jenkins newbie).
regards,
Colin
On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
> Thanks Kihwal for bringing this up.
>
> Seems related to:
>
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
> insight about the issue Kihwal described?
>
> Thanks.
>
> --Yongjun
>
>
> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
>> I am sure many of us have seen strange jenkins behavior out of the
>> precommit builds.
>>
>> - build artifacts missing
>> - serving build artifact belonging to another build. This also causes
>> wrong precommit results to be posted on the bug.
>> - etc.
>>
>> The latest one I saw is disappearance of the unit test stdout/stderr file
>> during a build. After a successful run of unit tests, the file vanished, so
>> the script could not cat it. It looked like another build process had
>> deleted it, while this build was in progress.
>>
>> It might have something to do with the fact that the patch-dir is set like
>> following:
>>
>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>> don't have access to the jenkins build configs or the build machines, so I
>> can't debug it further, but I think we need to take care of it sooner than
>> later. Can any one help?
>>
>> Kihwal
>>
Re: Erratic Jenkins behavior
Posted by "Colin P. McCabe" <cm...@apache.org>.
I'm sorry, I don't have any insight into this. With regard to
HADOOP-11084, I thought that $BUILD_URL would be unique for each
concurrent build, which would prevent build artifacts from getting
mixed up between jobs. Based on the value of PATCHPROCESS that Kihwal
posted, perhaps this is not the case? Perhaps someone can explain how
this is supposed to work (I am a Jenkins newbie).
regards,
Colin
On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang <yz...@cloudera.com> wrote:
> Thanks Kihwal for bringing this up.
>
> Seems related to:
>
> https://issues.apache.org/jira/browse/HADOOP-11084
>
> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
> insight about the issue Kihwal described?
>
> Thanks.
>
> --Yongjun
>
>
> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
> wrote:
>
>> I am sure many of us have seen strange jenkins behavior out of the
>> precommit builds.
>>
>> - build artifacts missing
>> - serving build artifact belonging to another build. This also causes
>> wrong precommit results to be posted on the bug.
>> - etc.
>>
>> The latest one I saw is disappearance of the unit test stdout/stderr file
>> during a build. After a successful run of unit tests, the file vanished, so
>> the script could not cat it. It looked like another build process had
>> deleted it, while this build was in progress.
>>
>> It might have something to do with the fact that the patch-dir is set like
>> following:
>>
>> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
>> don't have access to the jenkins build configs or the build machines, so I
>> can't debug it further, but I think we need to take care of it sooner than
>> later. Can any one help?
>>
>> Kihwal
>>
Re: Erratic Jenkins behavior
Posted by Yongjun Zhang <yz...@cloudera.com>.
Thanks Kihwal for bringing this up.
Seems related to:
https://issues.apache.org/jira/browse/HADOOP-11084
Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
insight about the issue Kihwal described?
Thanks.
--Yongjun
On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:
> I am sure many of us have seen strange jenkins behavior out of the
> precommit builds.
>
> - build artifacts missing
> - serving build artifact belonging to another build. This also causes
> wrong precommit results to be posted on the bug.
> - etc.
>
> The latest one I saw is disappearance of the unit test stdout/stderr file
> during a build. After a successful run of unit tests, the file vanished, so
> the script could not cat it. It looked like another build process had
> deleted it, while this build was in progress.
>
> It might have something to do with the fact that the patch-dir is set like
> following:
>
> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
> don't have access to the jenkins build configs or the build machines, so I
> can't debug it further, but I think we need to take care of it sooner than
> later. Can any one help?
>
> Kihwal
>
Re: Erratic Jenkins behavior
Posted by Yongjun Zhang <yz...@cloudera.com>.
Thanks Kihwal for bringing this up.
Seems related to:
https://issues.apache.org/jira/browse/HADOOP-11084
Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
insight about the issue Kihwal described?
Thanks.
--Yongjun
On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:
> I am sure many of us have seen strange jenkins behavior out of the
> precommit builds.
>
> - build artifacts missing
> - serving build artifact belonging to another build. This also causes
> wrong precommit results to be posted on the bug.
> - etc.
>
> The latest one I saw is disappearance of the unit test stdout/stderr file
> during a build. After a successful run of unit tests, the file vanished, so
> the script could not cat it. It looked like another build process had
> deleted it, while this build was in progress.
>
> It might have something to do with the fact that the patch-dir is set like
> following:
>
> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
> don't have access to the jenkins build configs or the build machines, so I
> can't debug it further, but I think we need to take care of it sooner than
> later. Can any one help?
>
> Kihwal
>
Re: Erratic Jenkins behavior
Posted by Yongjun Zhang <yz...@cloudera.com>.
Thanks Kihwal for bringing this up.
Seems related to:
https://issues.apache.org/jira/browse/HADOOP-11084
Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
insight about the issue Kihwal described?
Thanks.
--Yongjun
On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee <ki...@yahoo-inc.com.invalid>
wrote:
> I am sure many of us have seen strange jenkins behavior out of the
> precommit builds.
>
> - build artifacts missing
> - serving build artifact belonging to another build. This also causes
> wrong precommit results to be posted on the bug.
> - etc.
>
> The latest one I saw is disappearance of the unit test stdout/stderr file
> during a build. After a successful run of unit tests, the file vanished, so
> the script could not cat it. It looked like another build process had
> deleted it, while this build was in progress.
>
> It might have something to do with the fact that the patch-dir is set like
> following:
>
> PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/../patchprocessI
> don't have access to the jenkins build configs or the build machines, so I
> can't debug it further, but I think we need to take care of it sooner than
> later. Can any one help?
>
> Kihwal
>