You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "holdenk (via GitHub)" <gi...@apache.org> on 2024/02/28 00:58:17 UTC
[PR] [WIP] Add spark 35 [arrow-datafusion-comet]
holdenk opened a new pull request, #127:
URL: https://github.com/apache/arrow-datafusion-comet/pull/127
## Which issue does this PR close?
Adds Spark 3.5
## Rationale for this change
Spark 3.5 is the latest Spark release.
## What changes are included in this PR?
Adds Spark 3.5 support, this introduces a new per-spark version of shims as part of adding interfacing to the 3.5 APIs.
This has the advantage of static compile checks over the existing shim approach which does reflection for everything, but I can switch to the runtime reflection model for consistency if that's preferred.
## How are these changes tested?
WIP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Add spark 35 [arrow-datafusion-comet]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #127:
URL: https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1970294587
Some class and function has changed in spark 3.5, maybe we need some more shim classes to support both spark 3.5 and the versions before.
I has a similar commit to do this:
https://github.com/apache/arrow-datafusion-comet/commit/b038267eb5e6ddafccf84b0147760f6992f36077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Add spark 35 [arrow-datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #127:
URL: https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1971741645
FWIW I created https://github.com/apache/arrow-datafusion-comet/issues/140 to improve the current Spark shim handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Add spark 35 [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #127:
URL: https://github.com/apache/datafusion-comet/pull/127#issuecomment-2076485587
FYI https://github.com/apache/datafusion-comet/pull/318 is just merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [WIP] Add spark 35 [arrow-datafusion-comet]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #127:
URL: https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1970361538
Yup this adds some shims (there are existing shims too for 3.3 etc but they use reflection heavily).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Add spark 35 [arrow-datafusion-comet]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #127:
URL: https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1971775940
I love it :)
On Thu, Feb 29, 2024 at 10:45 AM Chao Sun ***@***.***> wrote:
> FWIW I created #140
> <https://github.com/apache/arrow-datafusion-comet/issues/140> to improve
> the current Spark shim handling.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1971741645>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAOT5NRC355APXEYYRFIADYV53MJAVCNFSM6AAAAABD5DERESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRG42DCNRUGU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org