You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/17 19:34:25 UTC
[GitHub] [arrow-ballista] andygrove opened a new pull request, #238: WIP: Add scheduler UI to docker
andygrove opened a new pull request, #238:
URL: https://github.com/apache/arrow-ballista/pull/238
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes https://github.com/apache/arrow-ballista/issues/11
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
I would like a Docker image containing the scheduler and the scheduler UI
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
- Create separate images for scheduler vs executor
- Include scheduler UI in scheduler image
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
No
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on a diff in pull request #238: WIP: Add scheduler UI to docker
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#discussion_r973622411
##########
dev/docker/ballista-scheduler.dockerfile:
##########
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG VERSION
+FROM apache/arrow-ballista:$VERSION
+
+# Scheduler UI
+RUN apt -y install nodejs npm nginx
+RUN npm install --global yarn
+RUN npm install --save node-releases
+RUN mkdir /tmp/ballista-ui
+WORKDIR /tmp/ballista-ui
+
+COPY ballista/ui/scheduler ./
+RUN yarn
+
+# TODO this fails with:
+# Error: Cannot find module '../../data/browsers'
+#RUN yarn build
+
+# TODO start nginx in background to serve the UI
Review Comment:
I need help with this part as I have zero experience with node or yarn :weary:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove merged pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove merged PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1250344114
@yahoNanJing @thinkharderdev @avantgardnerio This PR is now ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251070801
> Maybe better to add an example command to leverage these docker files.
I have now updated the relevant sections in the user guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on a diff in pull request #238: WIP: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#discussion_r973622778
##########
dev/docker/ballista-scheduler.dockerfile:
##########
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG VERSION
+FROM apache/arrow-ballista:$VERSION
+
+# Scheduler UI
+RUN apt -y install nodejs npm nginx
+RUN npm install --global yarn
+RUN npm install --save node-releases
+RUN mkdir /tmp/ballista-ui
+WORKDIR /tmp/ballista-ui
+
+COPY ballista/ui/scheduler ./
+RUN yarn
+
+# TODO this fails with:
+# Error: Cannot find module '../../data/browsers'
+#RUN yarn build
+
+# TODO start nginx in background to serve the UI
Review Comment:
cc @msathis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251170369
> documentation for using docker-compose
I don't understand `./dev/build-ballista-docker.sh`? Why not simply
```
docker-compose up --build
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251067774
> Pictures or it didn't happen :)
The UI says "scheduler hasn't started" now, but I saw this working previously. Once this PR is merged, I will start learning more about the UI code and see if I can start adding the new functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251158407
> > add an example command
>
> I personally favor `docker-compose.yml` because it can build the images and store all the runtime information like ports in a formal way.
Here is the user guide documentation for using docker-compose:
https://github.com/apache/arrow-ballista/blob/master/docs/source/user-guide/deployment/docker-compose.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251257843
We also have a helm chart I'd like to PR in as well to handle kubernetes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251256965
I was just in the process of making PRs out of all our changes now that DataFusion 12 was released. This is the PR: https://github.com/apache/arrow-ballista/pull/249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] yahoNanJing commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
yahoNanJing commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1250765506
Maybe better to add an example command to leverage these docker files. For example, like follows:
{code}
./dev/build-ballista-docker.sh
docker run -p 10.249.75.38:80:80 -p 10.249.75.38:50050:50050 -it apache/arrow-ballista-scheduler:0.8.0 --external-host 10.249.75.38 -s push-staged
docker run -p 10.249.75.38:50051:50051 -p 10.249.75.38:50052:50052 -it apache/arrow-ballista-executor:0.8.0 --external-host 10.249.75.38 --scheduler-host 10.249.75.38 -s push-staged
{code}
And it seems we need to fix something in file *executor_server.rs* to change the following part
{code}
let addr = format!(
"{}:{}",
executor_meta
.optional_host
.map(|h| match h {
OptionalHost::Host(host) => host,
})
.unwrap_or_else(|| String::from("0.0.0.0")),
executor_meta.grpc_port
);
{code}
to
{code}
let addr = format!(
"{}:{}",
"0.0.0.0",
executor_meta.grpc_port
);
{code}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251143634
> add an example command
I personally favor `docker-compose.yml` because it can build the images and store all the runtime information like ports in a formal way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1250390987
Pictures or it didn't happen :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251208089
> > documentation for using docker-compose
>
> I don't understand `./dev/build-ballista-docker.sh`? Why not simply
>
> ```
> docker-compose up --build
> ```
>
> ?
I'm not familiar with that feature. Do you want to PR to add that? We will still need a way to create the images for other uses, like k8s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251211094
> Also, why is out `docker-compose.yml` inside a `readme.md` vs just checked in?
This is for the user guide that will soon be published to the Arrow site. It is common practice for user guides to have sample yaml along with an explanation (see Spark's docs for example).
Ideally, we would have the yaml checked into the repo and tested and then generate that into the docs. I would like to get to that point eventually.
We do have one tested docker-compose.yaml checked into the `benchmarks` directory, and this is tested by the integration test script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on pull request #238: Add scheduler UI to ballista-scheduler Docker image
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #238:
URL: https://github.com/apache/arrow-ballista/pull/238#issuecomment-1251172694
Also, why is out `docker-compose.yml` inside a `readme.md` vs just checked in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org