You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/22 15:43:28 UTC
[I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
andygrove opened a new issue, #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144
**Describe the bug**
TPC-H q18 hangs with 16 executors x 8 cores in EKS. Executors show no activity after some time.
**To Reproduce**
TBD
**Expected behavior**
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5060878944
Executors do not use a memory pool by default (that is probably not good). I was not specifying `--memory-pool-size`, so we retry with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5060830800
I ran with pull based scheduling. Same error. I spent more time watching the logs and now see that executors are restarting. I suspect this is OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5049531241
Would changing to pull instead of push (I believe that's current one) make any difference ?
I saw those type of errors, can it be that executor gets too busy to send HB ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5048984769
Same for q9.
Saw this in scheduler log:
```
│ scheduler 2026-07-22T16:26:33.516100Z INFO tokio-rt-worker ThreadId(09) ballista_scheduler::state::executor_manager: Removing executor 0c71e449-aa94-4ed9-aae8-87bdae4a8a6f: Some("ACTIVE executor 0c71e449-aa94-4ed9-aae8-87bdae4a8a6f heartbeat timed out aft │
│ scheduler 2026-07-22T16:26:33.516101Z INFO tokio-rt-worker ThreadId(05) ballista_scheduler::state::executor_manager: Removing executor 3eca16d6-82ef-4456-82ad-2e964c052f62: Some("ACTIVE executor 3eca16d6-82ef-4456-82ad-2e964c052f62 heartbeat timed out aft │
│ scheduler 2026-07-22T16:26:33.516085Z INFO tokio-rt-worker ThreadId(06) ballista_scheduler::state::executor_manager: Removing executor 8e93fa02-1677-4871-8c12-6c1859bf5ef0: Some("ACTIVE executor 8e93fa02-1677-4871-8c12-6c1859bf5ef0 heartbeat timed out aft │
│ scheduler 2026-07-22T16:26:33.516094Z INFO tokio-rt-worker ThreadId(03) ballista_scheduler::state::executor_manager: Removing executor 9d0bcab9-e662-4f67-a0a0-7de26db4aaf5: Some("ACTIVE executor 9d0bcab9-e662-4f67-a0a0-7de26db4aaf5 heartbeat timed out aft │
│ scheduler 2026-07-22T16:26:33.516103Z INFO tokio-rt-worker ThreadId(08) ballista_scheduler::state::executor_manager: Removing executor c3c6a5b4-835d-4f1a-b1b0-6f90d43bcdc2: Some("ACTIVE executor c3c6a5b4-835d-4f1a-b1b0-6f90d43bcdc2 heartbeat timed out aft │
│ scheduler 2026-07-22T16:26:33.516115Z WARN tokio-rt-worker ThreadId(02) ballista_scheduler::state::aqe: Reset 8 tasks for running job/stage VkZQbC7/1 on lost Executor ed36558a-71c3-416a-ad2d-b4a47c89cae3 │
│ scheduler 2026-07-22T16:26:33.516104Z INFO tokio-rt-worker ThreadId(09) ballista_scheduler::state::executor_manager: Removing executor 57fe8e71-fb47-4ede-8ebd-e9c50b8d306f: Some("ACTIVE executor 57fe8e71-fb47-4ede-8ebd-e9c50b8d306f heartbeat timed out aft │
│ scheduler 2026-07-22T16:26:33.516126Z WARN tokio-rt-worker ThreadId(02) ballista_scheduler::state::aqe: Reset 8 tasks for running job/stage VkZQbC7/1 on lost Executor be83c07c-b61f-41c3-b6ab-305bb68d304f │
│ scheduler 2026-07-22T16:26:33.516146Z WARN tokio-rt-worker ThreadId(02) ballista_scheduler::state::aqe: Reset 8 tasks for running job/stage VkZQbC7/1 on lost Executor 784d05a8-fd1f-40d4-a5bf-ccf7a9a3be2d │
│ scheduler 2026-07-22T16:26:33.516161Z WARN tokio-rt-worker ThreadId(02) ballista_scheduler::state::aqe: Reset 8 tasks for running job/stage VkZQbC7/1 on lost Executor 8049c48e-5937-423a-8a3d-390a2ca7e4b2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2144: TPC-H q18 hangs with 16 executors x 8 cores in EKS
URL: https://github.com/apache/datafusion-ballista/issues/2144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5061183001
> How many partitions do you use for your run [@andygrove](https://github.com/andygrove) ?
Same as # of cores, so 16 x 8 = 128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5061299602
> So adding a memory pool will prevent OOM triggering spilling, probably. would increasing number of partitions make task smaller hence avoid spilling?
Potentially, yes, although if there is skew then some partitions will be much larger than others still.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5061042529
How many partitions do you use for your run @andygrove ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5052379352
FWIW we've seen similar issues in Spice (and we're on pull mode). We have fixes for some of the issues we've seen so far, I still need to work on upstreaming them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5060957862
Specifying a memory pool size fixed the issue 🎉
It seems like a bad experience for new users to hit this because the default is to use unbounded memory. I will look into what we can do to have a sensible default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5061259981
So adding a memory pool will prevent OOM triggering spilling, probably. would increasing number of partitions make task smaller hence avoid spilling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TPC-H q18 hangs with 16 executors x 8 cores in EKS [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2144:
URL: https://github.com/apache/datafusion-ballista/issues/2144#issuecomment-5059951250
https://github.com/apache/datafusion-ballista/pull/2159 did not help. q9 still shows heartbeat timeouts after 180s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org