You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Jason Wen <zh...@workday.com.INVALID> on 2021/03/23 23:53:26 UTC
Failed to build hadoop in our Bamboo agent
Hi team,
When we build Hadoop rpms on our bamboo agent, we got following error:
error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
The build works fine on our local machine, but hit this issue when running on our Bamboo agent.
The build command we use is:
./gradlew hadoop-pkg-ind -POS=centos-7 -Pprefix=trunk
We use master branch of bigtop repo when running the build command.
Can anyone shed light on the possible reason? /var/tmp/rpm-tmp.LARP0s looks like a file to be written to the build docker container.
Here are more build logs with the error:
build 23-Mar-2021 15:15:19 Provides: hadoop-libhdfs-devel = 3.2.2-1.el7 hadoop-libhdfs-devel(x86-64) = 3.2.2-1.el7
build 23-Mar-2021 15:15:19 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
build 23-Mar-2021 15:15:19 Processing files: hadoop-hdfs-fuse-3.2.2-1.el7.x86_64
build 23-Mar-2021 15:15:19 Finding Provides: /usr/lib/rpm/redhat/find-provides
build 23-Mar-2021 15:15:20 Provides: hadoop-hdfs-fuse = 3.2.2-1.el7 hadoop-hdfs-fuse(x86-64) = 3.2.2-1.el7
build 23-Mar-2021 15:15:20 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
build 23-Mar-2021 15:15:20 Processing files: hadoop-debuginfo-3.2.2-1.el7.x86_64
build 23-Mar-2021 15:15:20 Provides: hadoop-debuginfo = 3.2.2-1.el7 hadoop-debuginfo(x86-64) = 3.2.2-1.el7
build 23-Mar-2021 15:15:20 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
build 23-Mar-2021 15:15:20 Checking for unpackaged file(s): /usr/lib/rpm/check-files /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-3.2.2-1.el7.x86_64
error 23-Mar-2021 15:15:20 warning: Installed (but unpackaged) file(s) found:
error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/hadoop-user-functions.sh.example
error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/shellprofile.d/example.sh
error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/user_ec_policies.xml.template
error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/yarnservice-log4j.properties
error 23-Mar-2021 15:15:20 /usr/lib/hadoop/libexec/hadoop-layout.sh.example
build 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49 error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
build 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49 File listed twice: /usr/lib/hadoop/logs
build 23-Mar-2021 15:15:49 RPM build errors:
error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-hdfs/cache
error 23-Mar-2021 15:15:49 File listed twice: /usr/lib/hadoop-yarn/bin/container-executor
error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-yarn/cache
error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-mapreduce/cache
error 23-Mar-2021 15:15:49 Installed (but unpackaged) file(s) found:
error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/hadoop-user-functions.sh.example
error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/shellprofile.d/example.sh
error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/user_ec_policies.xml.template
error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/yarnservice-log4j.properties
error 23-Mar-2021 15:15:49 /usr/lib/hadoop/libexec/hadoop-layout.sh.example
error 23-Mar-2021 15:15:49 Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
build 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49
build 23-Mar-2021 15:15:49 > Task :hadoop-rpm FAILED
error 23-Mar-2021 15:15:49 FAILURE: Build failed with an exception.
build 23-Mar-2021 15:15:49 :hadoop-rpm (Thread[Execution worker for ':' Thread 5,5,main]) completed. Took 42 mins 10.972 secs.
error 23-Mar-2021 15:15:49
build 23-Mar-2021 15:15:49 5 actionable tasks: 5 executed
error 23-Mar-2021 15:15:49 * Where:
error 23-Mar-2021 15:15:49 Script '/bigtop/packages.gradle' line: 507
error 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49 * What went wrong:
error 23-Mar-2021 15:15:49 Execution failed for task ':hadoop-rpm'.
error 23-Mar-2021 15:15:49 > Process 'command 'rpmbuild'' finished with non-zero exit value 1
error 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49 * Try:
error 23-Mar-2021 15:15:49 Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
error 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49 * Get more help at https://help.gradle.org
error 23-Mar-2021 15:15:49
error 23-Mar-2021 15:15:49 BUILD FAILED in 42m 33s
Thanks,
Jason
Re: [External Sender] Failed to build hadoop in our Bamboo agent
Posted by Jason Wen <zh...@workday.com.INVALID>.
Thanks Olaf.
We switch to another bamboo agent and then it works.
-Jason
On 3/24/21, 12:10 PM, "Olaf Flebbe" <of...@oflebbe.de> wrote:
> Am 24.03.2021 um 19:42 schrieb Jason Wen <zhenshan.wen@workday.com.INVALID <ma...@workday.com.INVALID>>:
>
> Thanks Masatake for sharing this.
> Do you know the possible reasons in the case of write failure besides 'No space left on device'?
>
> Regards,
> Jason
There might be quotas or resouce limits in place.
Olaf
Re: [External Sender] Failed to build hadoop in our Bamboo agent
Posted by Olaf Flebbe <of...@oflebbe.de>.
> Am 24.03.2021 um 19:42 schrieb Jason Wen <zhenshan.wen@workday.com.INVALID <ma...@workday.com.INVALID>>:
>
> Thanks Masatake for sharing this.
> Do you know the possible reasons in the case of write failure besides 'No space left on device'?
>
> Regards,
> Jason
There might be quotas or resouce limits in place.
Olaf
Re: [External Sender] Re: Failed to build hadoop in our Bamboo agent
Posted by Jason Wen <zh...@workday.com.INVALID>.
Thanks Masatake for sharing this.
Do you know the possible reasons in the case of write failure besides 'No space left on device'?
Regards,
Jason
On 3/24/21, 1:31 AM, "Masatake Iwasaki" <iw...@oss.nttdata.co.jp> wrote:
Hi Jason,
> ./gradlew hadoop-pkg-ind -POS=centos-7 -Pprefix=trunk
I could not reproduce the issue by running the same command on my EC2 instance.
> error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
There should be strerror message showing the cause after the ":"
(e.g. "No space left on device").
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rpm-2Dsoftware-2Dmanagement_rpm_blob_rpm-2D4.11.3-2Drelease_build_pack.c-23L193-2DL195&d=DwICaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=iLmekl5bHiQsDH5Kb6fv_7zJRvoXIClF-1ttNrSRs1I&s=LyO0YSdE6FUyUoqYeZJugckKj-fw37hCfBDpXDP-9-s&e=
Empty strerror string implies that errno is not set.
I think it could be the case if `write` can not write nb bytes in one pass
while I can not suggest configuration to fix.
Masatake Iwasaki
On 2021/03/24 8:53, Jason Wen wrote:
> Hi team,
>
> When we build Hadoop rpms on our bamboo agent, we got following error:
> error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
>
> The build works fine on our local machine, but hit this issue when running on our Bamboo agent.
> The build command we use is:
> ./gradlew hadoop-pkg-ind -POS=centos-7 -Pprefix=trunk
> We use master branch of bigtop repo when running the build command.
>
> Can anyone shed light on the possible reason? /var/tmp/rpm-tmp.LARP0s looks like a file to be written to the build docker container.
>
> Here are more build logs with the error:
> build 23-Mar-2021 15:15:19 Provides: hadoop-libhdfs-devel = 3.2.2-1.el7 hadoop-libhdfs-devel(x86-64) = 3.2.2-1.el7
> build 23-Mar-2021 15:15:19 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
> build 23-Mar-2021 15:15:19 Processing files: hadoop-hdfs-fuse-3.2.2-1.el7.x86_64
> build 23-Mar-2021 15:15:19 Finding Provides: /usr/lib/rpm/redhat/find-provides
> build 23-Mar-2021 15:15:20 Provides: hadoop-hdfs-fuse = 3.2.2-1.el7 hadoop-hdfs-fuse(x86-64) = 3.2.2-1.el7
> build 23-Mar-2021 15:15:20 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
> build 23-Mar-2021 15:15:20 Processing files: hadoop-debuginfo-3.2.2-1.el7.x86_64
> build 23-Mar-2021 15:15:20 Provides: hadoop-debuginfo = 3.2.2-1.el7 hadoop-debuginfo(x86-64) = 3.2.2-1.el7
> build 23-Mar-2021 15:15:20 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
> build 23-Mar-2021 15:15:20 Checking for unpackaged file(s): /usr/lib/rpm/check-files /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-3.2.2-1.el7.x86_64
> error 23-Mar-2021 15:15:20 warning: Installed (but unpackaged) file(s) found:
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/hadoop-user-functions.sh.example
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/shellprofile.d/example.sh
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/user_ec_policies.xml.template
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/yarnservice-log4j.properties
> error 23-Mar-2021 15:15:20 /usr/lib/hadoop/libexec/hadoop-layout.sh.example
> build 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
> build 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 File listed twice: /usr/lib/hadoop/logs
> build 23-Mar-2021 15:15:49 RPM build errors:
> error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-hdfs/cache
> error 23-Mar-2021 15:15:49 File listed twice: /usr/lib/hadoop-yarn/bin/container-executor
> error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-yarn/cache
> error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-mapreduce/cache
> error 23-Mar-2021 15:15:49 Installed (but unpackaged) file(s) found:
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/hadoop-user-functions.sh.example
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/shellprofile.d/example.sh
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/user_ec_policies.xml.template
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/yarnservice-log4j.properties
> error 23-Mar-2021 15:15:49 /usr/lib/hadoop/libexec/hadoop-layout.sh.example
> error 23-Mar-2021 15:15:49 Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
> build 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49
> build 23-Mar-2021 15:15:49 > Task :hadoop-rpm FAILED
> error 23-Mar-2021 15:15:49 FAILURE: Build failed with an exception.
> build 23-Mar-2021 15:15:49 :hadoop-rpm (Thread[Execution worker for ':' Thread 5,5,main]) completed. Took 42 mins 10.972 secs.
> error 23-Mar-2021 15:15:49
> build 23-Mar-2021 15:15:49 5 actionable tasks: 5 executed
> error 23-Mar-2021 15:15:49 * Where:
> error 23-Mar-2021 15:15:49 Script '/bigtop/packages.gradle' line: 507
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 * What went wrong:
> error 23-Mar-2021 15:15:49 Execution failed for task ':hadoop-rpm'.
> error 23-Mar-2021 15:15:49 > Process 'command 'rpmbuild'' finished with non-zero exit value 1
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 * Try:
> error 23-Mar-2021 15:15:49 Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 * Get more help at https://urldefense.proofpoint.com/v2/url?u=https-3A__help.gradle.org&d=DwICaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=iLmekl5bHiQsDH5Kb6fv_7zJRvoXIClF-1ttNrSRs1I&s=Tgl1Fhxp-ShzwwGXIniEEh0VPku1DV0fNquuT-dKmgs&e=
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 BUILD FAILED in 42m 33s
>
> Thanks,
> Jason
>
Re: Failed to build hadoop in our Bamboo agent
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Jason,
> ./gradlew hadoop-pkg-ind -POS=centos-7 -Pprefix=trunk
I could not reproduce the issue by running the same command on my EC2 instance.
> error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
There should be strerror message showing the cause after the ":"
(e.g. "No space left on device").
https://github.com/rpm-software-management/rpm/blob/rpm-4.11.3-release/build/pack.c#L193-L195
Empty strerror string implies that errno is not set.
I think it could be the case if `write` can not write nb bytes in one pass
while I can not suggest configuration to fix.
Masatake Iwasaki
On 2021/03/24 8:53, Jason Wen wrote:
> Hi team,
>
> When we build Hadoop rpms on our bamboo agent, we got following error:
> error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
>
> The build works fine on our local machine, but hit this issue when running on our Bamboo agent.
> The build command we use is:
> ./gradlew hadoop-pkg-ind -POS=centos-7 -Pprefix=trunk
> We use master branch of bigtop repo when running the build command.
>
> Can anyone shed light on the possible reason? /var/tmp/rpm-tmp.LARP0s looks like a file to be written to the build docker container.
>
> Here are more build logs with the error:
> build 23-Mar-2021 15:15:19 Provides: hadoop-libhdfs-devel = 3.2.2-1.el7 hadoop-libhdfs-devel(x86-64) = 3.2.2-1.el7
> build 23-Mar-2021 15:15:19 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
> build 23-Mar-2021 15:15:19 Processing files: hadoop-hdfs-fuse-3.2.2-1.el7.x86_64
> build 23-Mar-2021 15:15:19 Finding Provides: /usr/lib/rpm/redhat/find-provides
> build 23-Mar-2021 15:15:20 Provides: hadoop-hdfs-fuse = 3.2.2-1.el7 hadoop-hdfs-fuse(x86-64) = 3.2.2-1.el7
> build 23-Mar-2021 15:15:20 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
> build 23-Mar-2021 15:15:20 Processing files: hadoop-debuginfo-3.2.2-1.el7.x86_64
> build 23-Mar-2021 15:15:20 Provides: hadoop-debuginfo = 3.2.2-1.el7 hadoop-debuginfo(x86-64) = 3.2.2-1.el7
> build 23-Mar-2021 15:15:20 Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
> build 23-Mar-2021 15:15:20 Checking for unpackaged file(s): /usr/lib/rpm/check-files /bigtop/build/hadoop/rpm/BUILDROOT/hadoop-3.2.2-1.el7.x86_64
> error 23-Mar-2021 15:15:20 warning: Installed (but unpackaged) file(s) found:
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/hadoop-user-functions.sh.example
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/shellprofile.d/example.sh
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/user_ec_policies.xml.template
> error 23-Mar-2021 15:15:20 /etc/hadoop/conf.empty/yarnservice-log4j.properties
> error 23-Mar-2021 15:15:20 /usr/lib/hadoop/libexec/hadoop-layout.sh.example
> build 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 error: Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
> build 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 File listed twice: /usr/lib/hadoop/logs
> build 23-Mar-2021 15:15:49 RPM build errors:
> error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-hdfs/cache
> error 23-Mar-2021 15:15:49 File listed twice: /usr/lib/hadoop-yarn/bin/container-executor
> error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-yarn/cache
> error 23-Mar-2021 15:15:49 File listed twice: /var/lib/hadoop-mapreduce/cache
> error 23-Mar-2021 15:15:49 Installed (but unpackaged) file(s) found:
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/hadoop-user-functions.sh.example
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/shellprofile.d/example.sh
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/user_ec_policies.xml.template
> error 23-Mar-2021 15:15:49 /etc/hadoop/conf.empty/yarnservice-log4j.properties
> error 23-Mar-2021 15:15:49 /usr/lib/hadoop/libexec/hadoop-layout.sh.example
> error 23-Mar-2021 15:15:49 Unable to write payload to /var/tmp/rpm-tmp.LARP0s:
> build 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49
> build 23-Mar-2021 15:15:49 > Task :hadoop-rpm FAILED
> error 23-Mar-2021 15:15:49 FAILURE: Build failed with an exception.
> build 23-Mar-2021 15:15:49 :hadoop-rpm (Thread[Execution worker for ':' Thread 5,5,main]) completed. Took 42 mins 10.972 secs.
> error 23-Mar-2021 15:15:49
> build 23-Mar-2021 15:15:49 5 actionable tasks: 5 executed
> error 23-Mar-2021 15:15:49 * Where:
> error 23-Mar-2021 15:15:49 Script '/bigtop/packages.gradle' line: 507
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 * What went wrong:
> error 23-Mar-2021 15:15:49 Execution failed for task ':hadoop-rpm'.
> error 23-Mar-2021 15:15:49 > Process 'command 'rpmbuild'' finished with non-zero exit value 1
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 * Try:
> error 23-Mar-2021 15:15:49 Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 * Get more help at https://help.gradle.org
> error 23-Mar-2021 15:15:49
> error 23-Mar-2021 15:15:49 BUILD FAILED in 42m 33s
>
> Thanks,
> Jason
>