You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by chinmaykolhatkar <gi...@git.apache.org> on 2017/03/21 13:15:21 UTC
[GitHub] bigtop pull request #188: BIGTOP-2706 Exclude hadoop dependencies from apex ...
GitHub user chinmaykolhatkar opened a pull request:
https://github.com/apache/bigtop/pull/188
BIGTOP-2706 Exclude hadoop dependencies from apex packaging for 3.5.0
While generating the rpm/deb, the package also bundles hadoop jars... Because of the clash of hadoop jars the apex cli fails to run.
Recently there has been change in min dependency of apex on hadoop from 2.2.0 to 2.6.0.
Proposal is to identify the min dependency of apex on hadoop at package time and exclude those jars in packaging.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chinmaykolhatkar/bigtop BIGTOP-2706_ApexPackageFix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/188.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #188
----
commit 409c408b2dde169a581c044c92a5a82868886a68
Author: Chinmay Kolhatkar <ch...@apache.org>
Date: 2017-03-21T13:13:05Z
BIGTOP-2706 Exclude hadoop dependencies from apex packaging for 3.5.0
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by c0s <gi...@git.apache.org>.
Github user c0s commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107769130
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
> I would also like to point out that the Apex package can be assembled without building Apex from source, by optionally using the official release
We actually _always_ build all the components from scratch. That's the policy aimed to guarantee the compatibility through the stack.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by rvs <gi...@git.apache.org>.
Github user rvs commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107508942
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
I think what @tweise and @c0s are saying is that you need to pass whatever flags are required to apex build so that it DOES pick up Hadoop version that is coming from Bigtop BOM. That's step #0
Then you can decide whether you want to exclude the JAR or not -- but it HAD to build with Hadoop that is enforced by Bigtop stack.
Does it make it clear?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bigtop/pull/188
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by rvs <gi...@git.apache.org>.
Github user rvs commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r108095450
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
@chinmaykolhatkar I've committed your change. I would like to make sure that we fixed the problem. What's the easiest way to run a test for this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107586606
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
Thanks @tweise @rvs @c0s . I'll update the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r110323508
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
@tweise Here is the jira: https://issues.apache.org/jira/browse/BIGTOP-2735
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Exclude hadoop dependencies from apex ...
Posted by c0s <gi...@git.apache.org>.
Github user c0s commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107272696
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
The very first material line of this file is to source `bigtop.bom` that contains the version of the components, included into the stack. Doing this grep you might effectively build Apex against a different version of Hadoop, hence introducing the inconsistency from the get go.
Any reason you don't want to build this component against Bigtop's Hadoop version? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107470382
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
For the purpose of this build, shouldn't it be the bigtop hadoop version, since Apex will run on top of it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r108093171
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
Agreed. I request this to be taken as an improvement for building apex component as a seperate Jira. Currently the apex component is broken and I believe that should be fixed. Let me know if this is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r108097463
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
Thanks @rvs Fastest way is to check if the deb/rpm package that there is no hadoop jar present. I'm writing package tests after for this for another jira (BIGTOP-2375).. That would make it even easier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107847630
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
@chinmaykolhatkar look at how the relevant jar files are filtered here: https://github.com/atrato/apex-cli-package/blob/master/pom.xml
It does not require the brittle exclusions list that you have below, because it works based on the apex-engine dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Exclude hadoop dependencies from apex ...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107335686
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
Apex min hadoop version is 2.6.0. I believe if I use hadoop version from bigtop.bom, that would give me 2.7.3. What I want to do here is, find out the min version of hadoop which apex uses and exclude those from packaging.
Hence done this way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop issue #188: BIGTOP-2706 Apex package does not run successfully after ...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on the issue:
https://github.com/apache/bigtop/pull/188
@c0s Change the title of PR to be same as that of jira.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107484387
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
I wanted to use the same. But that would mean excluding hadoop 2.7.3 dependencies and not 2.6.0 dependency.. We want to exclude the one that is referenced by apex. Hence done this way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r110193481
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
@chinmaykolhatkar did you create a JIRA for the build improvement?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop issue #188: BIGTOP-2706 Apex package does not run successfully after ...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on the issue:
https://github.com/apache/bigtop/pull/188
@c0s @rvs @tweise Made the changes. I had to add a diff for apex to compile successfully with 2.7.x.
The reason for the patch is captured here for reference: APEXCORE-679
In future releases of apex, we'll take care of jira in apex, that is when I can remove the diff.
Requesting to review and merge the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r108193720
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
}
# Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
# Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps -DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --
@chinmaykolhatkar agreed, that should be taken up separately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by chinmaykolhatkar <gi...@git.apache.org>.
Github user chinmaykolhatkar commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107836097
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
I'll be updating the PR where component is built from scratch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/bigtop/pull/188#discussion_r107581947
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -28,6 +28,9 @@ command -v mvn > /dev/null 2>&1 || {
# Build apex
mvn clean package -DskipTests "$@"
+#Find hadoop.version from pom.xml
+APEX_HADOOP_VERSION=$(grep -oPm1 "(?<=<hadoop.version>)[^<]+" pom.xml)
--- End diff --
@rsv that sums it up. Indeed it should be possible to pass the hadoop.version to the apex build.
I would also like to point out that the Apex package can be assembled without building Apex from source, by optionally using the official release artifacts from central: https://github.com/atrato/apex-cli-package/blob/master/pom.xml
That script also shows how the somewhat quirky dependency filtering pom can be simplified (and the stray legacy datatorrent references removed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---