You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Jagat Singh <ja...@gmail.com> on 2020/06/06 08:41:18 UTC
Bigtop Hadoop 3.2.1 error
Hi,
I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using
Ubuntu 18.04.
In the process of doing this, many questions came to my mind which I wanted
to learn about.
1) I changed the bom file and ran gradle hadoop-deb command. Is this the
correct process to create any deb or rpm?
2) In what state you will use patches present in folder common/src/hadoop/
, is it only usable when upstream system make additional changes after
making a release?
3) Many times the build fails in between and I had to clean up build and
output folder both and restart. What is the process to pick from where is
failed to save time?
4) Follow-up for 3), Building the hadoop and building deb for hadoop are
two different things, what command can be used to do one after the another
manually to save time in case one fails.
> Task :hadoop-deb FAILED
dpkg-source: warning: extracting unsigned source package
(hadoop_3.2.1-1.dsc)
dpkg-source: error: unpack target exists: hadoop-3.2.1
dpkg-source: info: extracting hadoop in hadoop-3.2.1
4) How does Bigtop ensures compatability between products, example Hive is
compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D
commands there is an override and version it takes from the bom file the
version information. Is this understanding correct? Do I still need to
change any maven pom file for any software to ensure compatibility?
5) Are there any more resources to learn about Bigtop other than
https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging. What
should I do next to learn more about Bigtop
6) Beside installing deb on my machine and playing with it, what else tests
are needed can be done to ensure it works as intended.
7) For Hadoop 3.1.2
I see there is a exit 1 in the install_hadoop.sh
My build fails around that, is there anyway to debug this. Why is this exit
1 ?
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438
+ '[' -e debian/tmp//usr/lib/hadoop-mapreduce//hadoop-annotations-3.2.1.jar
']'
+ exit 1
debian/rules:55: recipe for target 'override_dh_auto_install' failed
make[1]: *** [override_dh_auto_install] Error 1
make[1]: Leaving directory
'/home/jj/dev/code/open/bigtop/output/hadoop/hadoop-3.2.1'
debian/rules:27: recipe for target 'binary' failed
make: *** [binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned
exit status 2
debuild: fatal error at line 1152:
dpkg-buildpackage -rfakeroot -us -uc -ui -b
failed
Thanks in advance
Re: Bigtop Hadoop 3.2.1 error
Posted by Evans Ye <ev...@apache.org>.
> 6) Beside installing deb on my machine and playing with it, what else
tests are needed can be done to ensure it works as intended.
We've a testing framework that runs smoke tests pretty much in the
automatic way. Checkout [1].
Do aware that the deployment and testing feature is not fully covered
across all bigtop components. Checkout the supporting matrix[2]
[1]
https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
[2]
https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix
As Olaf said, just post your question here and we'd be happy to answer.
- Evans
Olaf Flebbe <of...@oflebbe.de> 於 2020年6月6日 週六 下午9:48寫道：
> Hi
>
> Thank you for you questions!
>
>
>
> Am 06.06.2020 um 10:41 schrieb Jagat Singh <ja...@gmail.com>:
>
> Hi,
>
> I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using
> Ubuntu 18.04.
>
> In the process of doing this, many questions came to my mind which I
> wanted to learn about.
>
> 1) I changed the bom file and ran gradle hadoop-deb command. Is this the
> correct process to create any deb or rpm?
>
>
> As a Bigtop User: no . You are not expected to roll your own Distribution .
> As a Developer: yes somehow, that’s usually the first try to upgrade a
> package.
> Since you already read the instructions: I like to stresst that you are
> supposed to use the docker containers (for git trunk
> bigtop/slaves:trunk-ubuntu-18.04 and run the gradle command within the
> container.
>
>
> 2) In what state you will use patches present in folder common/src/hadoop/
> , is it only usable when upstream system make additional changes after
> making a release?
>
>
> We try to use zero patches but if we find defects which are not fixed in
> the version we want to package, than we do patches. Or the build system is
> to be adapted to our needs… Usually we upstream fixes to the original
> project unless it is very bigtop specific. Most of the time we cherry-pick
> fixes from unreleased versions of the package.
>
> 3) Many times the build fails in between and I had to clean up build and
> output folder both and restart. What is the process to pick from where is
> failed to save time?
>
>
> No way, unfortunately. Neither deb nor rpm packaging have a way to resume
> where it failed.
>
> When I developed packaging I usually trying to do manually do the build,
> dependencies and try to automate stuff in the build scripts. This is very
> time consuming, indeed.
>
> Now you are hitting the ugly part. The upstream Hadoop project decided to
> rework all supporting start scripts, introduced different processes and
> frameworks with Hadoop3 , breaking our complete Hadoop setup.
>
> I did some initial work last year for Hadoop-3.1.2 in the "bigtop-alpha“
> git branch and dropped out of the project because of lack of time and
> changed priorities.
>
> If you like to invest into hadoop-3.2.1 I would recommend to look into
> this and merging with current master first.
>
>
> 4) Follow-up for 3), Building the hadoop and building deb for hadoop are
> two different things, what command can be used to do one after the another
> manually to save time in case one fails.
>
>
> Right, Building Hadoop should be straight forward, but packaging the new
> Hadoop 3.2 layout is hard, since Hadoop project decided to change to a
> monolithic approach. Without rewriting nearly all scripts it is not
> possible anymore to start daemons independently any more.
>
> I would recommend to do a manual installation first and start fresh with
> packaging the individual parts.
>
>
>
> > Task :hadoop-deb FAILED
> dpkg-source: warning: extracting unsigned source package
> (hadoop_3.2.1-1.dsc)
> dpkg-source: error: unpack target exists: hadoop-3.2.1
> dpkg-source: info: extracting hadoop in hadoop-3.2.1
>
> 4) How does Bigtop ensures compatability between products, example Hive is
> compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D
> commands there is an override and version it takes from the bom file the
> version information. Is this understanding correct? Do I still need to
> change any maven pom file for any software to ensure compatibility?
>
>
> You are right that is the way bigtop currently works. The crucial part is
> that it uses the local maven repository to pick up previous artifacts.
> (Dependency does "mvn install“ and mvn package of an package will pick up
> the previous compiled artifact from ~/.m2/repository ) .
>
> Additionally, some crucial parts like Hadoop or mapreduce jars are
> symlinked in packages so they are only installed once on the target system.
>
> I was proposing to not do this any more since we might break API’s that
> way (and we had that) . My approach in bigtop-alpha was to use the
> dependencies the devs chose and to accept different versions of a jar in
> the system, depending on the *network* API rather the *Java* API. That
> would allow us to compile packages independently.
>
>
> 5) Are there any more resources to learn about Bigtop other than
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging. What
> should I do next to learn more about Bigtop
>
>
> Best resources are questions. We will do our best to answer.
> It might help if you look into the jobs behind the scenes in
> ci.bigtop.apache.org . Please create an account there and I am happy to
> give you read access to the configuration of bigtop-trunk-packages for
> instance
>
>
>
> 6) Beside installing deb on my machine and playing with it, what else
> tests are needed can be done to ensure it works as intended.
>
>
> I think Evans can answer this best. We have docker deployments,
>
> 7) For Hadoop 3.1.2
>
> I see there is a exit 1 in the install_hadoop.sh
>
> My build fails around that, is there anyway to debug this. Why is this
> exit 1 ?
>
>
> I am assuming the jar file is simply not there any more, since Hadoop
> major version 3 changed a lot in the layout.
> AFAIK I worked around all these problems in the bigtop-alpha branch.
>
>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438
>
> + '[' -e
> debian/tmp//usr/lib/hadoop-mapreduce//hadoop-annotations-3.2.1.jar ']'
> + exit 1
>
> debian/rules:55: recipe for target 'override_dh_auto_install' failed
> make[1]: *** [override_dh_auto_install] Error 1
> make[1]: Leaving directory
> '/home/jj/dev/code/open/bigtop/output/hadoop/hadoop-3.2.1'
> debian/rules:27: recipe for target 'binary' failed
> make: *** [binary] Error 2
> dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned
> exit status 2
> debuild: fatal error at line 1152:
> dpkg-buildpackage -rfakeroot -us -uc -ui -b
> failed
>
> Thanks in advance
>
>
> The bigtop project would be very grateful if you submit pull requests for
> Hadoop 3.
>
> Best Olaf
>
>
>
>
Re: Bigtop Hadoop 3.2.1 error
Posted by Evans Ye <ev...@apache.org>.
> 6) Beside installing deb on my machine and playing with it, what else
tests are needed can be done to ensure it works as intended.
We've a testing framework that runs smoke tests pretty much in the
automatic way. Checkout [1].
Do aware that the deployment and testing feature is not fully covered
across all bigtop components. Checkout the supporting matrix[2]
[1]
https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
[2]
https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix
As Olaf said, just post your question here and we'd be happy to answer.
- Evans
Olaf Flebbe <of...@oflebbe.de> 於 2020年6月6日 週六 下午9:48寫道：
> Hi
>
> Thank you for you questions!
>
>
>
> Am 06.06.2020 um 10:41 schrieb Jagat Singh <ja...@gmail.com>:
>
> Hi,
>
> I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using
> Ubuntu 18.04.
>
> In the process of doing this, many questions came to my mind which I
> wanted to learn about.
>
> 1) I changed the bom file and ran gradle hadoop-deb command. Is this the
> correct process to create any deb or rpm?
>
>
> As a Bigtop User: no . You are not expected to roll your own Distribution .
> As a Developer: yes somehow, that’s usually the first try to upgrade a
> package.
> Since you already read the instructions: I like to stresst that you are
> supposed to use the docker containers (for git trunk
> bigtop/slaves:trunk-ubuntu-18.04 and run the gradle command within the
> container.
>
>
> 2) In what state you will use patches present in folder common/src/hadoop/
> , is it only usable when upstream system make additional changes after
> making a release?
>
>
> We try to use zero patches but if we find defects which are not fixed in
> the version we want to package, than we do patches. Or the build system is
> to be adapted to our needs… Usually we upstream fixes to the original
> project unless it is very bigtop specific. Most of the time we cherry-pick
> fixes from unreleased versions of the package.
>
> 3) Many times the build fails in between and I had to clean up build and
> output folder both and restart. What is the process to pick from where is
> failed to save time?
>
>
> No way, unfortunately. Neither deb nor rpm packaging have a way to resume
> where it failed.
>
> When I developed packaging I usually trying to do manually do the build,
> dependencies and try to automate stuff in the build scripts. This is very
> time consuming, indeed.
>
> Now you are hitting the ugly part. The upstream Hadoop project decided to
> rework all supporting start scripts, introduced different processes and
> frameworks with Hadoop3 , breaking our complete Hadoop setup.
>
> I did some initial work last year for Hadoop-3.1.2 in the "bigtop-alpha“
> git branch and dropped out of the project because of lack of time and
> changed priorities.
>
> If you like to invest into hadoop-3.2.1 I would recommend to look into
> this and merging with current master first.
>
>
> 4) Follow-up for 3), Building the hadoop and building deb for hadoop are
> two different things, what command can be used to do one after the another
> manually to save time in case one fails.
>
>
> Right, Building Hadoop should be straight forward, but packaging the new
> Hadoop 3.2 layout is hard, since Hadoop project decided to change to a
> monolithic approach. Without rewriting nearly all scripts it is not
> possible anymore to start daemons independently any more.
>
> I would recommend to do a manual installation first and start fresh with
> packaging the individual parts.
>
>
>
> > Task :hadoop-deb FAILED
> dpkg-source: warning: extracting unsigned source package
> (hadoop_3.2.1-1.dsc)
> dpkg-source: error: unpack target exists: hadoop-3.2.1
> dpkg-source: info: extracting hadoop in hadoop-3.2.1
>
> 4) How does Bigtop ensures compatability between products, example Hive is
> compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D
> commands there is an override and version it takes from the bom file the
> version information. Is this understanding correct? Do I still need to
> change any maven pom file for any software to ensure compatibility?
>
>
> You are right that is the way bigtop currently works. The crucial part is
> that it uses the local maven repository to pick up previous artifacts.
> (Dependency does "mvn install“ and mvn package of an package will pick up
> the previous compiled artifact from ~/.m2/repository ) .
>
> Additionally, some crucial parts like Hadoop or mapreduce jars are
> symlinked in packages so they are only installed once on the target system.
>
> I was proposing to not do this any more since we might break API’s that
> way (and we had that) . My approach in bigtop-alpha was to use the
> dependencies the devs chose and to accept different versions of a jar in
> the system, depending on the *network* API rather the *Java* API. That
> would allow us to compile packages independently.
>
>
> 5) Are there any more resources to learn about Bigtop other than
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging. What
> should I do next to learn more about Bigtop
>
>
> Best resources are questions. We will do our best to answer.
> It might help if you look into the jobs behind the scenes in
> ci.bigtop.apache.org . Please create an account there and I am happy to
> give you read access to the configuration of bigtop-trunk-packages for
> instance
>
>
>
> 6) Beside installing deb on my machine and playing with it, what else
> tests are needed can be done to ensure it works as intended.
>
>
> I think Evans can answer this best. We have docker deployments,
>
> 7) For Hadoop 3.1.2
>
> I see there is a exit 1 in the install_hadoop.sh
>
> My build fails around that, is there anyway to debug this. Why is this
> exit 1 ?
>
>
> I am assuming the jar file is simply not there any more, since Hadoop
> major version 3 changed a lot in the layout.
> AFAIK I worked around all these problems in the bigtop-alpha branch.
>
>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438
>
> + '[' -e
> debian/tmp//usr/lib/hadoop-mapreduce//hadoop-annotations-3.2.1.jar ']'
> + exit 1
>
> debian/rules:55: recipe for target 'override_dh_auto_install' failed
> make[1]: *** [override_dh_auto_install] Error 1
> make[1]: Leaving directory
> '/home/jj/dev/code/open/bigtop/output/hadoop/hadoop-3.2.1'
> debian/rules:27: recipe for target 'binary' failed
> make: *** [binary] Error 2
> dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned
> exit status 2
> debuild: fatal error at line 1152:
> dpkg-buildpackage -rfakeroot -us -uc -ui -b
> failed
>
> Thanks in advance
>
>
> The bigtop project would be very grateful if you submit pull requests for
> Hadoop 3.
>
> Best Olaf
>
>
>
>
Re: Bigtop Hadoop 3.2.1 error
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi
Thank you for you questions!
> Am 06.06.2020 um 10:41 schrieb Jagat Singh <ja...@gmail.com>:
>
> Hi,
>
> I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using Ubuntu 18.04.
>
> In the process of doing this, many questions came to my mind which I wanted to learn about.
>
> 1) I changed the bom file and ran gradle hadoop-deb command. Is this the correct process to create any deb or rpm?
As a Bigtop User: no . You are not expected to roll your own Distribution .
As a Developer: yes somehow, that’s usually the first try to upgrade a package.
Since you already read the instructions: I like to stresst that you are supposed to use the docker containers (for git trunk
bigtop/slaves:trunk-ubuntu-18.04 and run the gradle command within the container.
> 2) In what state you will use patches present in folder common/src/hadoop/ , is it only usable when upstream system make additional changes after making a release?
We try to use zero patches but if we find defects which are not fixed in the version we want to package, than we do patches. Or the build system is to be adapted to our needs… Usually we upstream fixes to the original project unless it is very bigtop specific. Most of the time we cherry-pick fixes from unreleased versions of the package.
> 3) Many times the build fails in between and I had to clean up build and output folder both and restart. What is the process to pick from where is failed to save time?
No way, unfortunately. Neither deb nor rpm packaging have a way to resume where it failed.
When I developed packaging I usually trying to do manually do the build, dependencies and try to automate stuff in the build scripts. This is very time consuming, indeed.
Now you are hitting the ugly part. The upstream Hadoop project decided to rework all supporting start scripts, introduced different processes and frameworks with Hadoop3 , breaking our complete Hadoop setup.
I did some initial work last year for Hadoop-3.1.2 in the "bigtop-alpha“ git branch and dropped out of the project because of lack of time and changed priorities.
If you like to invest into hadoop-3.2.1 I would recommend to look into this and merging with current master first.
> 4) Follow-up for 3), Building the hadoop and building deb for hadoop are two different things, what command can be used to do one after the another manually to save time in case one fails.
>
Right, Building Hadoop should be straight forward, but packaging the new Hadoop 3.2 layout is hard, since Hadoop project decided to change to a monolithic approach. Without rewriting nearly all scripts it is not possible anymore to start daemons independently any more.
I would recommend to do a manual installation first and start fresh with packaging the individual parts.
> > Task :hadoop-deb FAILED
> dpkg-source: warning: extracting unsigned source package (hadoop_3.2.1-1.dsc)
> dpkg-source: error: unpack target exists: hadoop-3.2.1
> dpkg-source: info: extracting hadoop in hadoop-3.2.1
>
> 4) How does Bigtop ensures compatability between products, example Hive is compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D commands there is an override and version it takes from the bom file the version information. Is this understanding correct? Do I still need to change any maven pom file for any software to ensure compatibility?
You are right that is the way bigtop currently works. The crucial part is that it uses the local maven repository to pick up previous artifacts. (Dependency does "mvn install“ and mvn package of an package will pick up the previous compiled artifact from ~/.m2/repository ) .
Additionally, some crucial parts like Hadoop or mapreduce jars are symlinked in packages so they are only installed once on the target system.
I was proposing to not do this any more since we might break API’s that way (and we had that) . My approach in bigtop-alpha was to use the dependencies the devs chose and to accept different versions of a jar in the system, depending on the *network* API rather the *Java* API. That would allow us to compile packages independently.
> 5) Are there any more resources to learn about Bigtop other than https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging <https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging>. What should I do next to learn more about Bigtop
Best resources are questions. We will do our best to answer.
It might help if you look into the jobs behind the scenes in ci.bigtop.apache.org <http://ci.bigtop.apache.org/> . Please create an account there and I am happy to give you read access to the configuration of bigtop-trunk-packages for instance
> 6) Beside installing deb on my machine and playing with it, what else tests are needed can be done to ensure it works as intended.
I think Evans can answer this best. We have docker deployments,
> 7) For Hadoop 3.1.2
>
> I see there is a exit 1 in the install_hadoop.sh
>
> My build fails around that, is there anyway to debug this. Why is this exit 1 ?
I am assuming the jar file is simply not there any more, since Hadoop major version 3 changed a lot in the layout.
AFAIK I worked around all these problems in the bigtop-alpha branch.
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438 <https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438>
>
> + '[' -e debian/tmp//usr/lib/hadoop-mapreduce//hadoop-annotations-3.2.1.jar ']'
> + exit 1
> debian/rules:55: recipe for target 'override_dh_auto_install' failed
> make[1]: *** [override_dh_auto_install] Error 1
> make[1]: Leaving directory '/home/jj/dev/code/open/bigtop/output/hadoop/hadoop-3.2.1'
> debian/rules:27: recipe for target 'binary' failed
> make: *** [binary] Error 2
> dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
> debuild: fatal error at line 1152:
> dpkg-buildpackage -rfakeroot -us -uc -ui -b
> failed
>
> Thanks in advance
>
The bigtop project would be very grateful if you submit pull requests for Hadoop 3.
Best Olaf
>
Re: Bigtop Hadoop 3.2.1 error
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi
Thank you for you questions!
> Am 06.06.2020 um 10:41 schrieb Jagat Singh <ja...@gmail.com>:
>
> Hi,
>
> I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using Ubuntu 18.04.
>
> In the process of doing this, many questions came to my mind which I wanted to learn about.
>
> 1) I changed the bom file and ran gradle hadoop-deb command. Is this the correct process to create any deb or rpm?
As a Bigtop User: no . You are not expected to roll your own Distribution .
As a Developer: yes somehow, that’s usually the first try to upgrade a package.
Since you already read the instructions: I like to stresst that you are supposed to use the docker containers (for git trunk
bigtop/slaves:trunk-ubuntu-18.04 and run the gradle command within the container.
> 2) In what state you will use patches present in folder common/src/hadoop/ , is it only usable when upstream system make additional changes after making a release?
We try to use zero patches but if we find defects which are not fixed in the version we want to package, than we do patches. Or the build system is to be adapted to our needs… Usually we upstream fixes to the original project unless it is very bigtop specific. Most of the time we cherry-pick fixes from unreleased versions of the package.
> 3) Many times the build fails in between and I had to clean up build and output folder both and restart. What is the process to pick from where is failed to save time?
No way, unfortunately. Neither deb nor rpm packaging have a way to resume where it failed.
When I developed packaging I usually trying to do manually do the build, dependencies and try to automate stuff in the build scripts. This is very time consuming, indeed.
Now you are hitting the ugly part. The upstream Hadoop project decided to rework all supporting start scripts, introduced different processes and frameworks with Hadoop3 , breaking our complete Hadoop setup.
I did some initial work last year for Hadoop-3.1.2 in the "bigtop-alpha“ git branch and dropped out of the project because of lack of time and changed priorities.
If you like to invest into hadoop-3.2.1 I would recommend to look into this and merging with current master first.
> 4) Follow-up for 3), Building the hadoop and building deb for hadoop are two different things, what command can be used to do one after the another manually to save time in case one fails.
>
Right, Building Hadoop should be straight forward, but packaging the new Hadoop 3.2 layout is hard, since Hadoop project decided to change to a monolithic approach. Without rewriting nearly all scripts it is not possible anymore to start daemons independently any more.
I would recommend to do a manual installation first and start fresh with packaging the individual parts.
> > Task :hadoop-deb FAILED
> dpkg-source: warning: extracting unsigned source package (hadoop_3.2.1-1.dsc)
> dpkg-source: error: unpack target exists: hadoop-3.2.1
> dpkg-source: info: extracting hadoop in hadoop-3.2.1
>
> 4) How does Bigtop ensures compatability between products, example Hive is compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D commands there is an override and version it takes from the bom file the version information. Is this understanding correct? Do I still need to change any maven pom file for any software to ensure compatibility?
You are right that is the way bigtop currently works. The crucial part is that it uses the local maven repository to pick up previous artifacts. (Dependency does "mvn install“ and mvn package of an package will pick up the previous compiled artifact from ~/.m2/repository ) .
Additionally, some crucial parts like Hadoop or mapreduce jars are symlinked in packages so they are only installed once on the target system.
I was proposing to not do this any more since we might break API’s that way (and we had that) . My approach in bigtop-alpha was to use the dependencies the devs chose and to accept different versions of a jar in the system, depending on the *network* API rather the *Java* API. That would allow us to compile packages independently.
> 5) Are there any more resources to learn about Bigtop other than https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging <https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging>. What should I do next to learn more about Bigtop
Best resources are questions. We will do our best to answer.
It might help if you look into the jobs behind the scenes in ci.bigtop.apache.org <http://ci.bigtop.apache.org/> . Please create an account there and I am happy to give you read access to the configuration of bigtop-trunk-packages for instance
> 6) Beside installing deb on my machine and playing with it, what else tests are needed can be done to ensure it works as intended.
I think Evans can answer this best. We have docker deployments,
> 7) For Hadoop 3.1.2
>
> I see there is a exit 1 in the install_hadoop.sh
>
> My build fails around that, is there anyway to debug this. Why is this exit 1 ?
I am assuming the jar file is simply not there any more, since Hadoop major version 3 changed a lot in the layout.
AFAIK I worked around all these problems in the bigtop-alpha branch.
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438 <https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/install_hadoop.sh#L438>
>
> + '[' -e debian/tmp//usr/lib/hadoop-mapreduce//hadoop-annotations-3.2.1.jar ']'
> + exit 1
> debian/rules:55: recipe for target 'override_dh_auto_install' failed
> make[1]: *** [override_dh_auto_install] Error 1
> make[1]: Leaving directory '/home/jj/dev/code/open/bigtop/output/hadoop/hadoop-3.2.1'
> debian/rules:27: recipe for target 'binary' failed
> make: *** [binary] Error 2
> dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
> debuild: fatal error at line 1152:
> dpkg-buildpackage -rfakeroot -us -uc -ui -b
> failed
>
> Thanks in advance
>
The bigtop project would be very grateful if you submit pull requests for Hadoop 3.
Best Olaf
>