You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/30 00:54:31 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
TheNeuralBit opened a new pull request #11575:
URL: https://github.com/apache/beam/pull/11575
Adds a simple tests that sends the planner_name option through to the SQL expansion service, and verifies it worked with a ZetaSQL-specific query.
Includes changes from #11574
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-631628284
Run XVR_Flink PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #11575:
URL: https://github.com/apache/beam/pull/11575#discussion_r428198550
##########
File path: sdks/python/apache_beam/transforms/sql.py
##########
@@ -31,8 +31,15 @@
__all__ = ['SqlTransform']
+_SQL_PLANNERS = {
+ 'zetasql': 'org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner',
Review comment:
Yeah I think that makes sense. Pushed a commit that moves it into the Java side. The only caveat is I had to move the external transform code out of SqlTransform itself (and into the expansion-service package) so that we can reference ZetaSQL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-621549896
The changes for this PR alone are in https://github.com/apache/beam/pull/11575/commits/3cc5424ab4b56d62ab076e55035fa336ce5a7381
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-631666570
XVR Flink [passed](https://builds.apache.org/job/beam_PostCommit_XVR_Flink_PR/100] before the package-info run, so I won't do it again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-622533616
The change looks good to me, once all its prerequisites get in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit edited a comment on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit edited a comment on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-631666570
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-621987541
Run XVR_Flink PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #11575:
URL: https://github.com/apache/beam/pull/11575
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-630291142
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-631629222
I plan to merge when CI is green
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-631688815
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-621549390
Run XVR_Flink PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11575:
URL: https://github.com/apache/beam/pull/11575#discussion_r427485815
##########
File path: sdks/python/apache_beam/transforms/sql.py
##########
@@ -31,8 +31,15 @@
__all__ = ['SqlTransform']
+_SQL_PLANNERS = {
+ 'zetasql': 'org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner',
Review comment:
I wonder if it would make more sense for this mapping to live on the Java side, and to have this parameter by called "dialect" or similar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-628316519
I just backed out the changes that relied on pipeline options, and instead plumbed through the local planner class name configuration added in #11647. Could you take another look @robertwb?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11575: [BEAM-9699] Add test verifying we can use ZetaSQL in Python SqlTransform
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11575:
URL: https://github.com/apache/beam/pull/11575#issuecomment-621549122
R: @robertwb
cc: @mxm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org