You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2024/09/09 06:39:35 UTC
[I] [VL] Disable background IO threads by default [incubator-gluten]
zhztheplayer opened a new issue, #7164:
URL: https://github.com/apache/incubator-gluten/issues/7164
### Description
Background IO threads (typically used for split preloading) have been causing some locking issues since it's not aligned with Spark's threading model. For example: https://github.com/apache/incubator-gluten/issues/7161.
And it also causes slow runtime filter: https://github.com/facebookincubator/velox/issues/10173.
So we are exploring opportunity to disable it by default until these related issues be resolved. Meanwhile would like to here from users for thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Disable background IO threads by default [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 commented on issue #7164:
URL: https://github.com/apache/incubator-gluten/issues/7164#issuecomment-2337466084
Thanks, make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Disable background IO threads by default [incubator-gluten]
Posted by "zhli1142015 (via GitHub)" <gi...@apache.org>.
zhli1142015 commented on issue #7164:
URL: https://github.com/apache/incubator-gluten/issues/7164#issuecomment-2337384412
Hello @zhztheplayer ,
May I ask some details about the issue #7161, did this happend recently or it exists from the time when this flag is enabled by default? As I don't recall there is such issue before.
This feature would impact outr perf number, as remote stroage access takes time, this would help reduce it. Do we have any plan to fix the issues and re-enable it?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Disable background IO threads by default [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #7164: [VL] Disable background IO threads by default
URL: https://github.com/apache/incubator-gluten/issues/7164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Disable background IO threads by default [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7164:
URL: https://github.com/apache/incubator-gluten/issues/7164#issuecomment-2354235383
What's the fundamental solution? I'm thinking to allocate memory in task threads always. io thread can only allocate memory from global memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Disable background IO threads by default [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #7164:
URL: https://github.com/apache/incubator-gluten/issues/7164#issuecomment-2337462116
Hi @zhli1142015 , as mentioned offline, issue https://github.com/apache/incubator-gluten/issues/7161 is not a new one and should exist for long time. I assume your scenario is not memory-bound so IO threads could work stably without spilling.
Although PR https://github.com/apache/incubator-gluten/pull/7165 for this is issue filed, we can still be tracking on any possible solutions for https://github.com/apache/incubator-gluten/issues/7161 after the PR is merged. And I am also adding one [TPC benchmark job](https://github.com/apache/incubator-gluten/pull/7165/files#diff-8dbcf212005134fd84bfbfc125fe323196c75a9bc87f81b3484d52ab1517d390R248-R284) for `iothreads=on` in the PR in case loosing test coverage. So I hope we can finally minimize the overall impacts on your use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org