You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "hokiegeek2 (via GitHub)" <gi...@apache.org> on 2023/01/23 19:30:04 UTC
[GitHub] [arrow-ballista] hokiegeek2 opened a new issue, #625: jobs from python client not showing up in Scheduler UI
hokiegeek2 opened a new issue, #625:
URL: https://github.com/apache/arrow-ballista/issues/625
**Describe the bug**
I am not totally sure this is a bug. The key thing is that I'm concerned that the ballista requests being submitted from my Python shell are not being distributed across my four-node cluster. I see the executors in the ui:

the ui is running here:

But I don't see any jobs:

I also don't see anything in the logs, so not sure if I am missing a config somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] hokiegeek2 closed issue #625: jobs from python client not showing up in Scheduler UI
Posted by "hokiegeek2 (via GitHub)" <gi...@apache.org>.
hokiegeek2 closed issue #625: jobs from python client not showing up in Scheduler UI
URL: https://github.com/apache/arrow-ballista/issues/625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] hokiegeek2 commented on issue #625: jobs from python client not showing up in Scheduler UI
Posted by "hokiegeek2 (via GitHub)" <gi...@apache.org>.
hokiegeek2 commented on issue #625:
URL: https://github.com/apache/arrow-ballista/issues/625#issuecomment-1401653590
@andygrove closing this issue, thanks for the clarification (and reminder about #595
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] hokiegeek2 commented on issue #625: jobs from python client not showing up in Scheduler UI
Posted by "hokiegeek2 (via GitHub)" <gi...@apache.org>.
hokiegeek2 commented on issue #625:
URL: https://github.com/apache/arrow-ballista/issues/625#issuecomment-1401563134
@andygrove thanks for the follow-up! On a hunch I did just that yesterday and got things working in terms of the sql API part:

so basically with BallistaContext the means to get a BallistaDataFrame is to execute a query against the sql API, works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on issue #625: jobs from python client not showing up in Scheduler UI
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #625:
URL: https://github.com/apache/arrow-ballista/issues/625#issuecomment-1400986935
You need to use BallistaContext rather than SessionContext. See https://github.com/apache/arrow-ballista/issues/595 for more details. Basically you are running thee query in DataFusion in the Python process and not in the Ballista cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org