You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/05/27 09:39:31 UTC

[GitHub] [bigtop] yoda-mon opened a new pull request, #907: [WIP] BIGTOP-3687: Add kubernetes libraries and related files

yoda-mon opened a new pull request, #907:
URL: https://github.com/apache/bigtop/pull/907

   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
   -->
   
   ### Description of PR
   Add kubernetes related libraries to spark package.
   
   ### How was this patch tested?
   1. Check the libraries are included
   
   ```sh
   
   # On CentOS7
   $ ./gradlew spark-pkg 
   $ rpm -qlp output/spark/noarch/spark-core-3.1.2-1.el7.noarch.rpm| grep kube
   /usr/lib/spark/jars/kubernetes-client-4.12.0.jar
   /usr/lib/spark/jars/kubernetes-model-admissionregistration-4.12.0.jar
   ...
   /usr/lib/spark/kubernetes/tests/worker_memory_check.py
   
   # On Ubuntu 18.04
   $ ./gradlew spark-pkg
   dpkg -c output/spark/spark-core_3.1.2-1_all.deb |grep kube
   -rw-r--r-- root/root    951167 2022-05-27 08:59 ./usr/lib/spark/jars/kubernetes-client-4.12.0.jar
   -rw-r--r-- root/root    836222 2022-05-27 08:59 ./usr/lib/spark/jars/kubernetes-model-admissionregistration-4.12.0.jar
   ...
   -rw-r--r-- root/root      1585 2022-05-27 08:59 ./usr/lib/spark/kubernetes/tests/worker_memory_check.py
   ```
   
   2.  Check whether it works
   
   ```
   # After Installing built RPMs on CentOS7 and running kubernetes by kind
   $ spark-shell  \
      --master k8s://https://127.0.0.1:37805  \
      --deploy-mode client  \
      --conf spark.executor.instances=1  \
      --conf spark.kubernetes.container.image=apache/spark:v3.1.3
   
   scala> val data = Seq(("Java", "20000"), ("Python", "100000"), ("Scala", "3000")).toDF.show()
   [Stage 0:>                                                          (0 + 1
   +------+------+
   |    _1|    _2|
   +------+------+
   |  Java| 20000|
   |Python|100000|
   | Scala|  3000|
   +------+------+
   ```
   
   3. Check it does not affect the existed features
   
   ```
   # On CentOS7
   ./gradlew \
     docker-provisioner \
     -POS=centos-7 \
     -Pnum_instances=3 \
     -Pmemory=8g \
     -Penable_local_repo \
     -Pstack="hdfs,yarn,mapreduce,spark" \
     -Psmoke_tests="spark" 
   
   # On Ubuntu 18.04
   ./gradlew \
     docker-provisioner \
     -POS=ubuntu-18.04 \
     -Pnum_instances=3 \
     -Pmemory=8g \
     -Penable_local_repo \
     -Pstack="hdfs,yarn,mapreduce,spark" \
     -Psmoke_tests="spark" 
   ```
   
   ### For code changes:
   
   - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
   - [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] iwasakims merged pull request #907: BIGTOP-3687: Add kubernetes libraries and related files

Posted by GitBox <gi...@apache.org>.
iwasakims merged PR #907:
URL: https://github.com/apache/bigtop/pull/907


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] yoda-mon commented on pull request #907: BIGTOP-3687: Add kubernetes libraries and related files

Posted by GitBox <gi...@apache.org>.
yoda-mon commented on PR #907:
URL: https://github.com/apache/bigtop/pull/907#issuecomment-1146978963

   Thank you for your review and merge.
   FYI, here is an example for separating packages.
   https://github.com/yoda-mon/bigtop/commit/0c03b7e59438ec1ac3cd326deeaedde71794c065


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] yoda-mon commented on pull request #907: BIGTOP-3687: Add kubernetes libraries and related files

Posted by GitBox <gi...@apache.org>.
yoda-mon commented on PR #907:
URL: https://github.com/apache/bigtop/pull/907#issuecomment-1139457863

   I considered separating the kubernetes package from the others. However the officially distributed Spark package now includes kubernetes libraries by default.
   https://spark.apache.org/downloads.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org