You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Matthew Jacobs <mj...@cloudera.com> on 2017/09/19 21:29:38 UTC
Re: jenkins.impala.io pre-existing workspace
Did anyone file a JIRA for this? I saw this again.
On Thu, Aug 31, 2017 at 1:36 PM, Jim Apple <jb...@cloudera.com> wrote:
> Also, to be clear, I don't have the cycles to lead the fix-the-cleanup
> task at the moment.
>
> On Wed, Aug 30, 2017 at 4:45 PM, Jim Apple <jb...@cloudera.com> wrote:
>> The workspace cleanup isn't working - see the last bit of any recent
>> ub1604 job: https://jenkins.impala.io/view/Utility/job/ubuntu-16.04-from-scratch/206/console
>>
>> 03:56:40.920 [WS-CLEANUP] Deleting project workspace...Cannot delete
>> workspace :remote file operation failed: /home/ubuntu at
>> hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
>> java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
>> (of a maximum of 3) waiting 0.1 sec between attempts.
>> 03:56:48.161 ERROR: Step ‘Delete workspace when build is done’ failed:
>> Cannot delete workspace: remote file operation failed: /home/ubuntu at
>> hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
>> java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
>> (of a maximum of 3) waiting 0.1 sec between attempts.
>>
>> The workspace is $HOME, so you can't just delete it without being root.
>>
>> This could be changed to
>>
>> 1. A post-build script to "rm -rf ~/*". This doesn't reset everything,
>> though - the job makes changes to other parts of the filesystem.
>>
>> 2. A post-build script to "sudo shutdown -h now" to make sure ec2
>> instances are not re-used. I'm not sure how Jenkins would feel about
>> this. :-)
>>
>> 3. A post-build script to move $HOME to some archived location on the
>> disk, to preserve debuggability.
>>
>> 4. A bash trap in the script to do one of the above.
>>
>> 5. Run the whole thing in a docker in the build machine, then delete
>> the container when the script is done. Or don't, if there's enough
>> disk space to not worry about that.
>>
>> 6. Do all of the work in a workspace inside $HOME. This would require
>> some changes to bootstrap_development.sh.
>>
>> #5 is the most hermetic, I'd guess.
>>
>> On Thu, Aug 24, 2017 at 8:29 AM, Michael Brown <mi...@cloudera.com> wrote:
>>> Looks like someone has done this.
>>>
>>> On Wed, Aug 23, 2017 at 8:16 PM, Alexander Behm <al...@cloudera.com>
>>> wrote:
>>>
>>>> Yes, let's please add the post-build action for sanity and consistency with
>>>> our other jobs.
>>>>
>>>> On Wed, Aug 23, 2017 at 7:42 PM, Tim Armstrong <ta...@cloudera.com>
>>>> wrote:
>>>>
>>>> > Maybe the workspace just got left in a weird state - I think in most
>>>> cases
>>>> > "git init" followed by checking out a branch and doing a clean would
>>>> work.
>>>> >
>>>> > Should we add the delete workspace post-build action?
>>>> >
>>>> > On Wed, Aug 23, 2017 at 5:32 PM, Michael Brown <mi...@cloudera.com>
>>>> wrote:
>>>> >
>>>> > > Not a known issue. I noticed ubuntu-16.04-from-scratch is not set to
>>>> > clean
>>>> > > up its workspace, and its config has not been touched since Aug 11. It
>>>> > > seems strange we only saw this now....
>>>> > >
>>>> > > On Wed, Aug 23, 2017 at 5:25 PM, Tim Armstrong <
>>>> tarmstrong@cloudera.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Is this a known problem? My job failed because the Impala repo
>>>> already
>>>> > > > existed on the machine:
>>>> > > >
>>>> > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/164/
>>>> > > >
>>>> > > > *23:00:24* + /usr/bin/git init /home/ubuntu/Impala*23:00:24*
>>>> > > > Reinitialized existing Git repository in /home/ubuntu/Impala/.git/
>>>> > > > ....
>>>> > > > *23:02:18* + for ITER in '$(seq 1 10)'*23:02:18* + echo 'ATTEMPT:
>>>> > > > 1'*23:02:18* ATTEMPT: 1*23:02:18* + /usr/bin/git checkout
>>>> > > > FETCH_HEAD*23:02:18* + cat
>>>> > > > /home/ubuntu/Impala/tmp.3tYBn0GUga*23:02:18* 23:02:18.712300
>>>> git.c:344
>>>> > > > trace: built-in: git 'checkout' 'FETCH_HEAD'*23:02:18*
>>>> > > > error: The following untracked working tree files would be
>>>> overwritten
>>>> > > > by checkout:*23:02:18* .clang-format*23:02:18*
>>>> > > > .clang-tidy*23:02:18*
>>>> > > > .gitignore*23:02:18* CMakeLists.txt*23:02:18*
>>>> > > > DISCLAIMER*23:02:18*
>>>> > > > EXPORT_CONTROL.md*23:02:18* LICENSE.txt*23:02:18*
>>>> > > > LOGS.md*23:02:18*
>>>> > > > NOTICE.txt*23:02:18* README.md*23:02:18*
>>>> > > > be/.gitignore*23:02:18*
>>>> > > > be/.impala.doxy*23:02:18* be/CMakeLists.txt*23:02:18*
>>>> > > > be/src/benchmarks/CMakeLists.txt*23:02:18*
>>>> > > > be/src/benchmarks/atod-benchmark.cc*23:02:18*
>>>> > > > be/src/benchmarks/atof-benchmark.cc*23:02:18*
>>>> > > > be/src/benchmarks/atoi-benchmark.cc*23:02:18*
>>>> > > > be/src/benchmarks/bit-packing-benchmark.cc*23:02:18*
>>>> > > > be/src/benchmarks/bitmap-benchmark.cc
>>>> > > > ...
>>>> > > >
>>>> > >
>>>> >
>>>>
Re: jenkins.impala.io pre-existing workspace
Posted by Matthew Jacobs <mj...@cloudera.com>.
https://issues.apache.org/jira/browse/IMPALA-5959
On Tue, Sep 19, 2017 at 2:54 PM, Jim Apple <jb...@cloudera.com> wrote:
> Nobody did.
>
> On Tue, Sep 19, 2017 at 2:29 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
>
>> Did anyone file a JIRA for this? I saw this again.
>>
>> On Thu, Aug 31, 2017 at 1:36 PM, Jim Apple <jb...@cloudera.com> wrote:
>> > Also, to be clear, I don't have the cycles to lead the fix-the-cleanup
>> > task at the moment.
>> >
>> > On Wed, Aug 30, 2017 at 4:45 PM, Jim Apple <jb...@cloudera.com> wrote:
>> >> The workspace cleanup isn't working - see the last bit of any recent
>> >> ub1604 job: https://jenkins.impala.io/view/Utility/job/ubuntu-16.04-
>> from-scratch/206/console
>> >>
>> >> 03:56:40.920 [WS-CLEANUP] Deleting project workspace...Cannot delete
>> >> workspace :remote file operation failed: /home/ubuntu at
>> >> hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
>> >> java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
>> >> (of a maximum of 3) waiting 0.1 sec between attempts.
>> >> 03:56:48.161 ERROR: Step ‘Delete workspace when build is done’ failed:
>> >> Cannot delete workspace: remote file operation failed: /home/ubuntu at
>> >> hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
>> >> java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
>> >> (of a maximum of 3) waiting 0.1 sec between attempts.
>> >>
>> >> The workspace is $HOME, so you can't just delete it without being root.
>> >>
>> >> This could be changed to
>> >>
>> >> 1. A post-build script to "rm -rf ~/*". This doesn't reset everything,
>> >> though - the job makes changes to other parts of the filesystem.
>> >>
>> >> 2. A post-build script to "sudo shutdown -h now" to make sure ec2
>> >> instances are not re-used. I'm not sure how Jenkins would feel about
>> >> this. :-)
>> >>
>> >> 3. A post-build script to move $HOME to some archived location on the
>> >> disk, to preserve debuggability.
>> >>
>> >> 4. A bash trap in the script to do one of the above.
>> >>
>> >> 5. Run the whole thing in a docker in the build machine, then delete
>> >> the container when the script is done. Or don't, if there's enough
>> >> disk space to not worry about that.
>> >>
>> >> 6. Do all of the work in a workspace inside $HOME. This would require
>> >> some changes to bootstrap_development.sh.
>> >>
>> >> #5 is the most hermetic, I'd guess.
>> >>
>> >> On Thu, Aug 24, 2017 at 8:29 AM, Michael Brown <mi...@cloudera.com>
>> wrote:
>> >>> Looks like someone has done this.
>> >>>
>> >>> On Wed, Aug 23, 2017 at 8:16 PM, Alexander Behm <
>> alex.behm@cloudera.com>
>> >>> wrote:
>> >>>
>> >>>> Yes, let's please add the post-build action for sanity and
>> consistency with
>> >>>> our other jobs.
>> >>>>
>> >>>> On Wed, Aug 23, 2017 at 7:42 PM, Tim Armstrong <
>> tarmstrong@cloudera.com>
>> >>>> wrote:
>> >>>>
>> >>>> > Maybe the workspace just got left in a weird state - I think in most
>> >>>> cases
>> >>>> > "git init" followed by checking out a branch and doing a clean would
>> >>>> work.
>> >>>> >
>> >>>> > Should we add the delete workspace post-build action?
>> >>>> >
>> >>>> > On Wed, Aug 23, 2017 at 5:32 PM, Michael Brown <mi...@cloudera.com>
>> >>>> wrote:
>> >>>> >
>> >>>> > > Not a known issue. I noticed ubuntu-16.04-from-scratch is not set
>> to
>> >>>> > clean
>> >>>> > > up its workspace, and its config has not been touched since Aug
>> 11. It
>> >>>> > > seems strange we only saw this now....
>> >>>> > >
>> >>>> > > On Wed, Aug 23, 2017 at 5:25 PM, Tim Armstrong <
>> >>>> tarmstrong@cloudera.com>
>> >>>> > > wrote:
>> >>>> > >
>> >>>> > > > Is this a known problem? My job failed because the Impala repo
>> >>>> already
>> >>>> > > > existed on the machine:
>> >>>> > > >
>> >>>> > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/164/
>> >>>> > > >
>> >>>> > > > *23:00:24* + /usr/bin/git init /home/ubuntu/Impala*23:00:24*
>> >>>> > > > Reinitialized existing Git repository in
>> /home/ubuntu/Impala/.git/
>> >>>> > > > ....
>> >>>> > > > *23:02:18* + for ITER in '$(seq 1 10)'*23:02:18* + echo
>> 'ATTEMPT:
>> >>>> > > > 1'*23:02:18* ATTEMPT: 1*23:02:18* + /usr/bin/git checkout
>> >>>> > > > FETCH_HEAD*23:02:18* + cat
>> >>>> > > > /home/ubuntu/Impala/tmp.3tYBn0GUga*23:02:18* 23:02:18.712300
>> >>>> git.c:344
>> >>>> > > > trace: built-in: git 'checkout'
>> 'FETCH_HEAD'*23:02:18*
>> >>>> > > > error: The following untracked working tree files would be
>> >>>> overwritten
>> >>>> > > > by checkout:*23:02:18* .clang-format*23:02:18*
>> >>>> > > > .clang-tidy*23:02:18*
>> >>>> > > > .gitignore*23:02:18* CMakeLists.txt*23:02:18*
>> >>>> > > > DISCLAIMER*23:02:18*
>> >>>> > > > EXPORT_CONTROL.md*23:02:18* LICENSE.txt*23:02:18*
>> >>>> > > > LOGS.md*23:02:18*
>> >>>> > > > NOTICE.txt*23:02:18* README.md*23:02:18*
>> >>>> > > > be/.gitignore*23:02:18*
>> >>>> > > > be/.impala.doxy*23:02:18*
>> be/CMakeLists.txt*23:02:18*
>> >>>> > > > be/src/benchmarks/CMakeLists.txt*23:02:18*
>> >>>> > > > be/src/benchmarks/atod-benchmark.cc*23:02:18*
>> >>>> > > > be/src/benchmarks/atof-benchmark.cc*23:02:18*
>> >>>> > > > be/src/benchmarks/atoi-benchmark.cc*23:02:18*
>> >>>> > > > be/src/benchmarks/bit-packing-benchmark.cc*23:02:18*
>> >>>> > > > be/src/benchmarks/bitmap-benchmark.cc
>> >>>> > > > ...
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>>
Re: jenkins.impala.io pre-existing workspace
Posted by Jim Apple <jb...@cloudera.com>.
Nobody did.
On Tue, Sep 19, 2017 at 2:29 PM, Matthew Jacobs <mj...@cloudera.com> wrote:
> Did anyone file a JIRA for this? I saw this again.
>
> On Thu, Aug 31, 2017 at 1:36 PM, Jim Apple <jb...@cloudera.com> wrote:
> > Also, to be clear, I don't have the cycles to lead the fix-the-cleanup
> > task at the moment.
> >
> > On Wed, Aug 30, 2017 at 4:45 PM, Jim Apple <jb...@cloudera.com> wrote:
> >> The workspace cleanup isn't working - see the last bit of any recent
> >> ub1604 job: https://jenkins.impala.io/view/Utility/job/ubuntu-16.04-
> from-scratch/206/console
> >>
> >> 03:56:40.920 [WS-CLEANUP] Deleting project workspace...Cannot delete
> >> workspace :remote file operation failed: /home/ubuntu at
> >> hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
> >> java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
> >> (of a maximum of 3) waiting 0.1 sec between attempts.
> >> 03:56:48.161 ERROR: Step ‘Delete workspace when build is done’ failed:
> >> Cannot delete workspace: remote file operation failed: /home/ubuntu at
> >> hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c):
> >> java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times
> >> (of a maximum of 3) waiting 0.1 sec between attempts.
> >>
> >> The workspace is $HOME, so you can't just delete it without being root.
> >>
> >> This could be changed to
> >>
> >> 1. A post-build script to "rm -rf ~/*". This doesn't reset everything,
> >> though - the job makes changes to other parts of the filesystem.
> >>
> >> 2. A post-build script to "sudo shutdown -h now" to make sure ec2
> >> instances are not re-used. I'm not sure how Jenkins would feel about
> >> this. :-)
> >>
> >> 3. A post-build script to move $HOME to some archived location on the
> >> disk, to preserve debuggability.
> >>
> >> 4. A bash trap in the script to do one of the above.
> >>
> >> 5. Run the whole thing in a docker in the build machine, then delete
> >> the container when the script is done. Or don't, if there's enough
> >> disk space to not worry about that.
> >>
> >> 6. Do all of the work in a workspace inside $HOME. This would require
> >> some changes to bootstrap_development.sh.
> >>
> >> #5 is the most hermetic, I'd guess.
> >>
> >> On Thu, Aug 24, 2017 at 8:29 AM, Michael Brown <mi...@cloudera.com>
> wrote:
> >>> Looks like someone has done this.
> >>>
> >>> On Wed, Aug 23, 2017 at 8:16 PM, Alexander Behm <
> alex.behm@cloudera.com>
> >>> wrote:
> >>>
> >>>> Yes, let's please add the post-build action for sanity and
> consistency with
> >>>> our other jobs.
> >>>>
> >>>> On Wed, Aug 23, 2017 at 7:42 PM, Tim Armstrong <
> tarmstrong@cloudera.com>
> >>>> wrote:
> >>>>
> >>>> > Maybe the workspace just got left in a weird state - I think in most
> >>>> cases
> >>>> > "git init" followed by checking out a branch and doing a clean would
> >>>> work.
> >>>> >
> >>>> > Should we add the delete workspace post-build action?
> >>>> >
> >>>> > On Wed, Aug 23, 2017 at 5:32 PM, Michael Brown <mi...@cloudera.com>
> >>>> wrote:
> >>>> >
> >>>> > > Not a known issue. I noticed ubuntu-16.04-from-scratch is not set
> to
> >>>> > clean
> >>>> > > up its workspace, and its config has not been touched since Aug
> 11. It
> >>>> > > seems strange we only saw this now....
> >>>> > >
> >>>> > > On Wed, Aug 23, 2017 at 5:25 PM, Tim Armstrong <
> >>>> tarmstrong@cloudera.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Is this a known problem? My job failed because the Impala repo
> >>>> already
> >>>> > > > existed on the machine:
> >>>> > > >
> >>>> > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/164/
> >>>> > > >
> >>>> > > > *23:00:24* + /usr/bin/git init /home/ubuntu/Impala*23:00:24*
> >>>> > > > Reinitialized existing Git repository in
> /home/ubuntu/Impala/.git/
> >>>> > > > ....
> >>>> > > > *23:02:18* + for ITER in '$(seq 1 10)'*23:02:18* + echo
> 'ATTEMPT:
> >>>> > > > 1'*23:02:18* ATTEMPT: 1*23:02:18* + /usr/bin/git checkout
> >>>> > > > FETCH_HEAD*23:02:18* + cat
> >>>> > > > /home/ubuntu/Impala/tmp.3tYBn0GUga*23:02:18* 23:02:18.712300
> >>>> git.c:344
> >>>> > > > trace: built-in: git 'checkout'
> 'FETCH_HEAD'*23:02:18*
> >>>> > > > error: The following untracked working tree files would be
> >>>> overwritten
> >>>> > > > by checkout:*23:02:18* .clang-format*23:02:18*
> >>>> > > > .clang-tidy*23:02:18*
> >>>> > > > .gitignore*23:02:18* CMakeLists.txt*23:02:18*
> >>>> > > > DISCLAIMER*23:02:18*
> >>>> > > > EXPORT_CONTROL.md*23:02:18* LICENSE.txt*23:02:18*
> >>>> > > > LOGS.md*23:02:18*
> >>>> > > > NOTICE.txt*23:02:18* README.md*23:02:18*
> >>>> > > > be/.gitignore*23:02:18*
> >>>> > > > be/.impala.doxy*23:02:18*
> be/CMakeLists.txt*23:02:18*
> >>>> > > > be/src/benchmarks/CMakeLists.txt*23:02:18*
> >>>> > > > be/src/benchmarks/atod-benchmark.cc*23:02:18*
> >>>> > > > be/src/benchmarks/atof-benchmark.cc*23:02:18*
> >>>> > > > be/src/benchmarks/atoi-benchmark.cc*23:02:18*
> >>>> > > > be/src/benchmarks/bit-packing-benchmark.cc*23:02:18*
> >>>> > > > be/src/benchmarks/bitmap-benchmark.cc
> >>>> > > > ...
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
>