You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/02/14 08:20:03 UTC

[GitHub] [bigtop] tsudukim opened a new pull request #597: BIGTOP-3165. Bump Spark to 2.4.4.

tsudukim opened a new pull request #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597
 
 
   Bump Spark to 2.4.4.
   
   `./docker-hadoop.sh -c 1 -s` passed on my local with the following configuration.
   
   ```
   docker:
           memory_limit: "16g"
           image:  "bigtop/puppet:trunk-ubuntu-16.04"
   
   repo: "http://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/$(ARCH)"
   distro: debian
   components: [hdfs, yarn, spark]
   enable_local_repo: true
   smoke_test_components: [spark]
   ```
   
   Since Spark src tarball doesn't inculde license files for binary as of 2.4, I added the the same binary license files to be included in bigtop binary.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] evans-ye commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
evans-ye commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-586547392
 
 
   I've no much knowledge here so CMIIW.
   Why we need to add these LICENSE files? I though it should be able to obtain from spark's code release.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-586564757
 
 
   @evans-ye Thank you for your review! I agree with you, license files should be able to obtain from spark's source but they aren't.
   Spark community devided license files between source licenses and binary licenses from Spark 2.4. Spark github repository contains both license files, but as I wrote in `bigtop-packages/src/common/spark/install_spark.sh` src tarball no longer contains the license files for binary distribution at lease at Spark 2.4. Src tarball only have the license files only for source distribution.
   See https://issues.apache.org/jira/browse/SPARK-24654
   In Bigtop, we build Spark from src tarball so we have several choices here. For example, 1) Just remove every license files from Bigtop Spark binary. 2) Get license files from Spark github repository and include them into Bigtop Spark binary. I thought 1 is not acceptable as an Apache project so I PRed 2, but please let me know if you have a better way.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-589457826
 
 
   Thanks a lot @evans-ye . The concern you wrote make sense and 2 seems to be a reasonable solution for us. So I will fix this PR by adding some safety message and changing the target version to 2.4.5 instead of 2.4.4.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] sekikn commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
sekikn commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-598658205
 
 
   @tsudukim I merged your PR manually rather than using GitHub's merge button so that I edited the commit summary (2.4.4 -> 2.4.5). So I'm going to close this PR. Thank you for the contribution!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] evans-ye commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
evans-ye commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-587599883
 
 
   Thanks for the detailed answer. And I'm glad that the change is captured instead of sliped through.
   
   So what I was worry about is that the LICENSE files can be outdated if we maintain our own copy.
   How about we add some safety to make sure that in any future upgrade, the licence file can be updated?
   
   Possible solutions:
   1. Auto fetch licence file when building. The downside is this require network dependency, which may not be possible in company intranet.
   2. Add a comment "remember to fetch new licence-binary" in common/spark alongside with the spark version defined in bigtop.bom. So when someone change it, the comment should be read.
   3. Just get it in and hope that we can remember it in future upgrades ;)
   
   I slightly prefer 2..., any input from your end?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] sekikn commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
sekikn commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-586166811
 
 
   Basically the PR looks good to me but I'd like to upgrade Spark to 2.4.5, because 2.4.4 doesn't work with the recent k8s versions. Could you address it?
   https://issues.apache.org/jira/browse/SPARK-28921

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-589535257
 
 
   Changed the version to Spark 2.4.5 and added safety comments in both bigtop.bom and `bigtop-packages/src/common/spark/install_spark.sh` . Could you check it again, please? @evans-ye @sekikn

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
tsudukim commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-586178748
 
 
   @sekikn Thank you for your comment. I will update this PR to 2.4.5 after BOM is updated on BIGTOP-3123.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] evans-ye edited a comment on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
evans-ye edited a comment on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-587599883
 
 
   Thanks for the detailed answer. And I'm glad that the change is captured in your PR. Thanks!
   
   So what I was worry about is that the LICENSE files can be outdated if we maintain our own copy.
   How about we add some safety to make sure that in any future upgrade, the licence file can be updated?
   
   Possible solutions:
   1. Auto fetch licence file when building. The downside is this require network dependency, which may not be possible in company intranet.
   2. Add a comment "remember to fetch new licence-binary" in common/spark alongside with the spark version defined in bigtop.bom. So when someone change it, the comment should be read.
   3. Just get it in and hope that we can remember it in future upgrades ;)
   
   I slightly prefer 2..., any input from your end?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] sekikn closed pull request #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
sekikn closed pull request #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [bigtop] sekikn commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.

Posted by GitBox <gi...@apache.org>.
sekikn commented on issue #597: BIGTOP-3165. Bump Spark to 2.4.4.
URL: https://github.com/apache/bigtop/pull/597#issuecomment-598655796
 
 
   +1, I confirmed that it was successfully built on CentOS 8, Debian 10, and Ubuntu 18.04. Also, its smoke test using the built packages passed in an Ubuntu 16.04 container.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services