You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "EpsilonPrime (via GitHub)" <gi...@apache.org> on 2024/08/14 06:56:26 UTC
[PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
EpsilonPrime opened a new pull request, #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835
Adds the join types currently defined in Substrait to the Gluten copy. This is one of a vast set of changes aimed at reducing the differences from the official version in hope that one day there would be no differences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2306378883
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2305981963
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "EpsilonPrime (via GitHub)" <gi...@apache.org>.
EpsilonPrime commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2295715712
> @EpsilonPrime Before we decide to take effort on migrating to mainstream Substrait, we may do some study on how much it could help us on next moves.
>
> Thing is if we unfork Substrait, we should use it for supporting more backends in Gluten. Otherwise we get less benefit than cost.
>
> Do you happen to know the progress of Substrait integration of some projects, for example, DuckDB, Arrow and Datafusion, if Gluten decide to add support for these libraries, are their Substrait consumer implementations reliable enough for us to use?
DuckDB has the best Substrait support of the three. Datafusion has a few issues which I'm hoping are addressed in their next release. Acero is in maintenance mode but has a working implementation but it's very strict about what it accepts.
Other benefits include tools which run on Substrait (like the validator and text plan format) which aren't really being used by Gluten at the moment.
The Spark proposal to move the Gluten communication logic (which may or may not have included Substrait) there was the reason I started looking into this effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2333378742
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2294884513
@EpsilonPrime Before we decide to take effort on migrating to mainstream Substrait, we may do some study on how much it could help us on next moves.
Thing is if we unfork Substrait, we should use it for supporting more backends in Gluten. Otherwise we get no benefit but only cost.
Do you happen to know the progress of Substrait integration of some projects, for example, DuckDB, Arrow and Datafusion, if Gluten decide to add support for these libraries, are their Substrait consumer implementations reliable enough for us to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][CORE] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2335019914
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2287990806
https://github.com/apache/incubator-gluten/issues/6834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2290442094
May add a new type for existence join ? existence join is transformed to left semi join
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2289850842
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2290645935
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][CORE] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2333550227
It seems CH CI failing at
```
15:08:17 /home/jenkins/agent/workspace/gluten/gluten-ci/gluten/cpp-ch/local-engine/Parser/JoinRelParser.cpp:272:80: error: no member named 'JoinRel_JoinType_JOIN_TYPE_ANTI' in namespace 'substrait'
15:08:17 272 | if (join_opt_info.is_null_aware_anti_join && join.type() == substrait::JoinRel_JoinType_JOIN_TYPE_ANTI)
15:08:17 | ~~~~~~~~~~~^
15:08:17 1 error generated.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2305982831
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2287991132
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "EpsilonPrime (via GitHub)" <gi...@apache.org>.
EpsilonPrime commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2290397082
There are also physical relations in Substrait. The approach I am taking at unforking is to slowly chip away at the differences so we can swap the main instance in seamlessly. A swap of everything at once seems unlikely so I'm tackling what I can -- especially if there are minor code changes related to each small change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "EpsilonPrime (via GitHub)" <gi...@apache.org>.
EpsilonPrime commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2290449317
> May add a new type for existence join ? existence join is transformed to left semi join at present existence join is supported both at ch and velox now.
We are in the process of adding new join types to the specification at the moment so adding another is definitely a possibility. (Currently I have a pr out to add mark join.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2290346381
@FelixYBW, @baibaichen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2290376947
I suspect how it actually helps if we make such changes.
For example both Spark and Velox doesn't have a join type `Right Anti` at the moment (despite that whether they will add it in future or not). So it may look confusing to have it defined in the interchange protocol.
As commented in https://github.com/apache/incubator-gluten/pull/6833#issuecomment-2290309584, physical plans would vary more than logical plans in a project like Gluten. We are already following both Spark's and Velox/CH's plan protocols. It could have chance to start messing things up if we strictly follow another one in the middle layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2295474341
Gluten modifications to Substrait: https://github.com/apache/incubator-gluten/blob/main/docs/developers/SubstraitModifications.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2295162561
> As commented in [#6833 (comment)](https://github.com/apache/incubator-gluten/pull/6833#issuecomment-2290309584), physical plans would vary more than logical plans in Gluten. We are already having to follow both Spark's and Velox/CH's plan protocols. It could have chance to start messing things up if we strictly follow another one in the middle layer.
we should have the same protocol in long term, substrait should be able to fit all requirements from different frameworks, libraries and accelerators. It's substrait's goal. But now for Gluten it's not urgent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2295163184
@JkSelf @rui-mo Do you happen to have the list of our modifications in Glute's substrait? I remember we have several pending PRs to upstream substrait but there is no active review, so we paused there.
@EpsilonPrime you may start to review the pending PRs in substrait if we have. Add the missing features Gluten needs to upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][core] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "EpsilonPrime (via GitHub)" <gi...@apache.org>.
EpsilonPrime commented on PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835#issuecomment-2295699525
> @JkSelf @rui-mo Do you happen to have the list of our modifications in Glute's substrait? I remember we have several pending PRs to upstream substrait but there is no active review, so we paused there.
>
> @EpsilonPrime you may start to review the pending PRs in substrait if we have. Add the missing features Gluten needs to upstream.
All of the pending changes were merged last year (about the time I became a reviewer). I have been also merging changes upstream (for instance the equivalent of TextReadOptions was added last week).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-6834][CORE] feat: add other join types from the official Substrait [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #6835:
URL: https://github.com/apache/incubator-gluten/pull/6835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org