You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ntjohnson1 (via GitHub)" <gi...@apache.org> on 2026/04/28 15:44:51 UTC
[PR] Allow pickling PyExpr [datafusion-python]
ntjohnson1 opened a new pull request, #1517:
URL: https://github.com/apache/datafusion-python/pull/1517
# Which issue does this PR close?
Allows holding a pyexpr when spawing python parallelism. Will file an issue. Currently just exploring.
Closes #.
# Rationale for this change
Parallelism.
# What changes are included in this PR?
This does change how global context is managed to treat it more like a singleton so we can assume that is the right thing to pull when unpacking the expr.
# Are there any user-facing changes?
Not really, besides adding new capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow pickling PyExpr [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on PR #1517:
URL: https://github.com/apache/datafusion-python/pull/1517#issuecomment-4525744805
Closing this since Tim's approach is more comprehensive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow pickling PyExpr [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on code in PR #1517:
URL: https://github.com/apache/datafusion-python/pull/1517#discussion_r3213102693
##########
.github/workflows/test.yml:
##########
@@ -29,6 +29,7 @@ env:
jobs:
test-matrix:
runs-on: ubuntu-latest
+ timeout-minutes: 30
Review Comment:
Right now things pass in 4 minutes but I accidentally committed a change that deadlocked and would have run for 6 hours if I didn't push new commits. Seems like a nice guard to footguns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow pickling PyExpr [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 closed pull request #1517: Allow pickling PyExpr
URL: https://github.com/apache/datafusion-python/pull/1517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow pickling PyExpr [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1517:
URL: https://github.com/apache/datafusion-python/pull/1517#issuecomment-4450612630
I'm testing out an approach that wouldn't use the `set_global_ctx()` requirement and instead use worker's init hooks to set state. It adds a tiny bit more work on setting up, but I don't think it's unreasonable. Also I'm going to add a codec that will allow for pickling of python written UDFs to be encoded/decoded so this will work over the wire.
It will probably be two PRs - one for the codec work first and then a follow on with the distribution/multiprocess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow pickling PyExpr [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on PR #1517:
URL: https://github.com/apache/datafusion-python/pull/1517#issuecomment-4412515731
Hmm I can either remove the multiprocessing test since the pickle stuff should be sufficient coverage or investigate why it's not hanging locally but is on CI later this weekend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org