You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/03 04:12:48 UTC
[GitHub] [spark] xuanyuanking opened a new pull request #29633: [SPARK-32782][SS] Refactory StreamingRelationV2 and move it to catalyst
xuanyuanking opened a new pull request #29633:
URL: https://github.com/apache/spark/pull/29633
### What changes were proposed in this pull request?
Move StreamingRelationV2 to the catalyst module and bind with the Table interface.
### Why are the changes needed?
Currently, the StreamingRelationV2 is bind with TableProvider. Since the V2 relation is not bound with `DataSource`, to make it more flexible and have better expansibility, it should be moved to the catalyst module and bound with the Table interface. We did a similar thing for DataSourceV2Relation.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29633: [SPARK-32782][SS] Refactory StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686241565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686267468
```
Does it couple with core module?
```
@HeartSaVioR Yes, the StreamingRelation couples with DataSource. You may find all the DataSource resolving work is in `FindDataSourceTable` so it's ok to put it in the core module. But for v2 relation, it's unbound with the specific data source, both `Table` and `TableProvider` is in the catalyst, so move it can benefit for our future extension on StreamingRelationV2.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686270291
In general, the v1 plan should still be sql/core as v1 API is there. The v2 plan should be in catalyst as v2 API is there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686299296
**[Test build #128226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128226/testReport)** for PR 29633 at commit [`1e24af5`](https://github.com/apache/spark/commit/1e24af571c75f15357e6b11e965983c3fb814c64).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class StreamingRelationV2(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686299791
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686264658
I'm not clear about the policy - if we would like to ensure the logical plan is placed in catalyst module, then why not moving StreamingRelation as well? Does it couple with core module?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686241240
**[Test build #128226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128226/testReport)** for PR 29633 at commit [`1e24af5`](https://github.com/apache/spark/commit/1e24af571c75f15357e6b11e965983c3fb814c64).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29633:
URL: https://github.com/apache/spark/pull/29633
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29633: [SPARK-32782][SS] Refactory StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686241565
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686297629
OK that sounds like the ideal place is in catalyst module, but we already put them to core module and they're exposed to the public already. Thanks for the explanation.
I wish we migrate to V2 sooner; looks like there're lots of TODO comments and built-in connectors look to be still V1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686298844
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29633: [SPARK-32782][SS] Refactory StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686241240
**[Test build #128226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128226/testReport)** for PR 29633 at commit [`1e24af5`](https://github.com/apache/spark/commit/1e24af571c75f15357e6b11e965983c3fb814c64).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686299799
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128226/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29633: [SPARK-32782][SS] Refactor StreamingRelationV2 and move it to catalyst
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29633:
URL: https://github.com/apache/spark/pull/29633#issuecomment-686299791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org