You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/09 17:07:45 UTC
[GitHub] [beam] robinyqiu opened a new pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
robinyqiu opened a new pull request #11957:
URL: https://github.com/apache/beam/pull/11957
R: @apilloud
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #11957:
URL: https://github.com/apache/beam/pull/11957#issuecomment-644318238
Because of the problems mentioned above, I did not change the implementation of `zetaSqlTypeToBeamFieldType`, instead I added another overload of it which does not depend on Beam `fieldType`. It will allow compliance test drive to call it (cl/316152825).
@apilloud Please check if this PR still LGTY.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu merged pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
robinyqiu merged pull request #11957:
URL: https://github.com/apache/beam/pull/11957
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu edited a comment on pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
robinyqiu edited a comment on pull request #11957:
URL: https://github.com/apache/beam/pull/11957#issuecomment-644318238
Because of the problems mentioned above, I did not change the implementation of `zetaSqlTypeToBeamFieldType`, instead I added another overload of it which does not depend on Beam `fieldType`. It will allow compliance test driver to call it (cl/316152825).
@apilloud Please check if this PR still LGTY.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] apilloud commented on pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
apilloud commented on pull request #11957:
URL: https://github.com/apache/beam/pull/11957#issuecomment-641631947
Thanks! We have a bunch of nullness issues, investigating some of those is how I discovered this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #11957:
URL: https://github.com/apache/beam/pull/11957#issuecomment-641627057
Well. Some tests are failing and I realized this turns out to be not that simple change. There are subtle differences between Beam schema and ZetaSQL struct that prevents this change (e.g. if ZetaSQL types are by default nullable whereas it is a label in Beam field type; and ZetaSQL struct allows fields with same name but Beam does not).
I will take more time to figure out what is the correct thing to do here. In the meanwhile, I will send out some other small cleanup PRs for review. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amaliujia commented on a change in pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #11957:
URL: https://github.com/apache/beam/pull/11957#discussion_r440590858
##########
File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUtils.java
##########
@@ -196,7 +196,55 @@ private static Value beamLogicalObjectToZetaSqlValue(Object object, String ident
}
}
- // Value conversion: ZetaSQL => Beam
+ // Type conversion: ZetaSQL => Beam
+ public static FieldType zetaSqlTypeToBeamFieldType(Type type) {
Review comment:
Out of curiosity: so the ZetaSQL type does not carry nullability? (otherwise seems the nullability of the type has lost in this conversion).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #11957:
URL: https://github.com/apache/beam/pull/11957#issuecomment-644299021
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #11957: [BEAM-10033] zetaSqlValueToJavaObject uses value.getType()
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #11957:
URL: https://github.com/apache/beam/pull/11957#discussion_r440591940
##########
File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUtils.java
##########
@@ -196,7 +196,55 @@ private static Value beamLogicalObjectToZetaSqlValue(Object object, String ident
}
}
- // Value conversion: ZetaSQL => Beam
+ // Type conversion: ZetaSQL => Beam
+ public static FieldType zetaSqlTypeToBeamFieldType(Type type) {
Review comment:
Yes. All ZetaSQL types are nullable. i.e. You can't define a ZetaSQL type that disallow null values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org