You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/23 09:07:12 UTC
[GitHub] [spark] tech4242 opened a new pull request #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
tech4242 opened a new pull request #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987
### What changes were proposed in this pull request?
I have changed a few paths in the Dockerfile that do not seem to reflect the structure of the project anymore. I have described this in detail in SPARK-31165.
### Why are the changes needed?
I think this is clear from above and the commit itself. If this is not
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Manual testing since the Dockerfile build command was not successful, so this mostly involved trial and error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27987: [SPARK-31165] Correcting
wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-607238730
@tech4242 do you want to further update?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-604296482
Thank you @dongjoon-hyun & @srowen I will work on this today/tomorrow and I'll have something ready on the weekend :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-603927838
Ya. I saw the same PRs before. This seems to make people confused. Some documentation may help us.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602670293
@srowen @yaooqinn thank you for your feedback!
I now ran it with `dev/make-distribution.sh -Pkubernetes` instead of `./build/mvn -Pkubernetes -DskipTests clean package` and it in fact does create a new folder in spark/dist/ where the structure is as described in the Dockerfile.
I propose that we mention this more explicitly in the documentation https://spark.apache.org/docs/latest/running-on-kubernetes.html to not run `./build/mvn -Pkubernetes -DskipTests clean package` because if you are new-ish to the ecosystem you might end up running that command instead of downloading the distribution or running the said bash script (regardless of what is mentioned in the Prerequisites about the Spark distribution, which I am myself guilty of reading over too quickly)
Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-603929739
BTW, https://spark.apache.org/docs/latest/running-on-kubernetes.html might be not a good place to have that. Please consider https://spark.apache.org/docs/latest/building-spark.html , too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27987: [SPARK-31165] Correcting
wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602629925
Yes, at least some of those paths look correct for the distribution structured, not the source tree, so I'd be surprised if this change works as intended. It's possible the kubernetes path is wrong. I'd just ask for more confirmation here, that you built the distro and ran this command and it failed for path X, and the real path is clearly Y now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27987: [SPARK-31165] Correcting
wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602727690
Probably a small section under Prerequisites about "How to Build" is fine, unless I'm missing that this is documented elsewhere already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602472275
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602511147
@yaooqinn thanks for reaching out! :) 2 questions from my side:
1) How is the `/kubernetes/ ` path for the resources (in the Dockerfile) ever going to be correct if one is supposed to run this from the root folder (as described in the Dockerfile comment) if all the files are under `/resource-managers/kubernetes/...` and if you e.g. build from source does the folder `/integration-tests/ `suddenly turn to `/tests/` ? I still feel like there are inconsistencies? I feel like your comment might only address the jars folder but for that I would need an answer to 2) below.
2) Do you mean that basically building with `./build/mvn -Pkubernetes -DskipTests clean package` first is pointless or not intended? If that is the case then the documentation should at least be updated or made more a bit easier to understand and I do think that the Dockerfile comment at the beginning of the file is also confusing.
Open to suggestions though - I am just trying to wrap my head around the intended behaviour of this Dockerfile and how it can possible run out of the box.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27987:
[SPARK-31165] Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602472275
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 edited a comment on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 edited a comment on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602511147
@yaooqinn thanks for reaching out! :) 2 questions from my side:
1) How is the `/kubernetes/ ` path for the resources (in the Dockerfile) ever going to be correct if one is supposed to run this from the root folder (as described in the Dockerfile comment) if all the files are under `/resource-managers/kubernetes/...` and if you e.g. build from source does the folder `/integration-tests/ `suddenly turn to `/tests/` ? I still feel like there are inconsistencies? I think your comment might only address the jars folder but for that I would need an answer to 2) below.
2) Do you mean that basically building with `./build/mvn -Pkubernetes -DskipTests clean package` first is pointless or not intended? If that is the case then the documentation should at least be updated or made a bit easier to understand and I do think that the Dockerfile comment at the beginning of the file is also confusing.
Open to suggestions though - I am just trying to wrap my head around the intended behaviour of this Dockerfile and how it can possible run out of the box.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 edited a comment on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 edited a comment on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602511147
@yaooqinn thanks for reaching out! :) 2 questions from my side:
1) How is the `/kubernetes/ ` path for the resources (in the Dockerfile) ever going to be correct if one is supposed to run this from the root folder (as described in the Dockerfile comment) if all the files are under `/resource-managers/kubernetes/...` and if you e.g. build from source does the folder `/integration-tests/ `suddenly turn to `/tests/` ? I still feel like there are inconsistencies? I think your comment might only address the jars folder but for that I would need an answer to 2) below.
2) Do you mean that basically building with `./build/mvn -Pkubernetes -DskipTests clean package` first is pointless or not intended? If that is the case then the documentation should at least be updated or made a bit easier to understand and I do think that the Dockerfile comment at the beginning of the file is also confusing.
Open to suggestions though - I am just trying to wrap my head around the intended behaviour of this Dockerfile and how it can run out of the box.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 edited a comment on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 edited a comment on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602511147
@yaooqinn thanks for reaching out! :) 2 questions from my side:
1) How is the `/kubernetes/ ` path for the resources (in the Dockerfile) ever going to be correct if one is supposed to run this from the root folder (as described in the Dockerfile comment) if all the files are under `/resource-managers/kubernetes/...` and if you e.g. build from source does the folder `/integration-tests/ `suddenly turn to `/tests/` ? I still feel like there are inconsistencies? I feel like your comment might only address the jars folder but for that I would need an answer to 2) below.
2) Do you mean that basically building with `./build/mvn -Pkubernetes -DskipTests clean package` first is pointless or not intended? If that is the case then the documentation should at least be updated or made a bit easier to understand and I do think that the Dockerfile comment at the beginning of the file is also confusing.
Open to suggestions though - I am just trying to wrap my head around the intended behaviour of this Dockerfile and how it can possible run out of the box.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tech4242 commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
tech4242 commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-603119872
@srowen sounds good! Would it be ok if I revert the change, squash the commits and make the document change here? Alternatively I'll create a new PR and link to this one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on issue #27987: [SPARK-31165] Correcting
wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
srowen commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-603355901
Sure just update this pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602526336
1. Building spark image from source code side, you may use:
```
bin/docker-image-tool.sh -r <repo> -t <tag> build
```
2. Building spark image form distribution you may need to run `dev/make-distribution.sh` with `-Pkubernetes` profile first, untar it, then run the script above under the root of that unzipped folder.
BTW, I am not a committer here, just giving my humble opinion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602493443
In fact, you need to download or build spark distribution first. The Dockerfile is not intended to work with spark source code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27987: [SPARK-31165]
Correcting wrong paths in Dockerfile
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27987: [SPARK-31165] Correcting wrong paths in Dockerfile
URL: https://github.com/apache/spark/pull/27987#issuecomment-602472823
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org