You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "xumingming (via GitHub)" <gi...@apache.org> on 2024/05/17 01:38:08 UTC
[PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
xumingming opened a new pull request, #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782
## What changes were proposed in this pull request?
If generator function's input is already Attribute reference, we omit the introduction of the ProjectExec.
(Fixes: \#5757)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116944578
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma merged PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116680656
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116473526
https://github.com/apache/incubator-gluten/issues/5757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2119732005
> spark-test-spark32-sl
I think it's in suite `GlutenSQLQueryTestSuite` and you could try modifying `supportedList` to run a single test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2123846934
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2124468494
@zhztheplayer `Gluten ClickHouse CI ` failed, it needs login, what's the username and password?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2120029327
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2116473765
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2124120614
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2126260510
All tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2124073766
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2119577687
@zhztheplayer Seems the test `transform.sql` in `spark-test-spark32-slow` failed, do you happen to know how to run this single test locally on my laptop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2119579345
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2126324207
cc @marin-ma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on code in PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#discussion_r1604487543
##########
gluten-core/src/test/scala/org/apache/gluten/execution/WholeStageTransformerSuite.scala:
##########
@@ -252,6 +252,34 @@ abstract class WholeStageTransformerSuite
assert(executedPlan.exists(plan => tag.runtimeClass.isInstance(plan)))
}
+ /**
+ * Check whether the executed plan of a dataframe contains the expected plan chain.
+ *
+ * @param df
+ * : the input dataframe.
+ * @param tag
+ * : class of the expected plan.
+ * @param childTag
+ * : class of the expected plan's child.
+ * @tparam T
+ * : type of the expected plan.
+ * @tparam PT
+ * : type of the expected plan's child.
+ */
+ def checkSparkOperatorChainMatch[T <: SparkPlan, PT <: SparkPlan](
Review Comment:
Good catch! Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2117523138
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#discussion_r1604381360
##########
gluten-core/src/test/scala/org/apache/gluten/execution/WholeStageTransformerSuite.scala:
##########
@@ -252,6 +252,34 @@ abstract class WholeStageTransformerSuite
assert(executedPlan.exists(plan => tag.runtimeClass.isInstance(plan)))
}
+ /**
+ * Check whether the executed plan of a dataframe contains the expected plan chain.
+ *
+ * @param df
+ * : the input dataframe.
+ * @param tag
+ * : class of the expected plan.
+ * @param childTag
+ * : class of the expected plan's child.
+ * @tparam T
+ * : type of the expected plan.
+ * @tparam PT
+ * : type of the expected plan's child.
+ */
+ def checkSparkOperatorChainMatch[T <: SparkPlan, PT <: SparkPlan](
Review Comment:
Should we use `T <: UnaryExecNode` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-5757][CORE] Remove unnecessary ProjectExecTransformer for Generate [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on PR #5782:
URL: https://github.com/apache/incubator-gluten/pull/5782#issuecomment-2124133602
@zhztheplayer Since https://github.com/apache/incubator-gluten/pull/5838 is merged, this PR is ready for review now.
Previous implementation always assume there is **dedicated** Project under Generate, now this is not true anymore. If there is a Project under Generate it could be a dedicated Project for Generate, or it also would be just a normal Project. So we added a metadata(injectedProject) in Substrait plan to tell us whether there is a dedicated Project under Generate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org