You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/02 18:58:50 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
TheNeuralBit opened a new pull request #13003:
URL: https://github.com/apache/beam/pull/13003
Adds an `@SchemaCaseFormat` that can be used to control the case format used in schema field names. For example:
```java
@DefaultSchema(JavaFieldSchema.class)
@SchemaCaseFormat(CaseFormat.LOWER_UNDERSCORE)
class MyClass {
public String user;
public int ageInYears;
@SchemaCaseFormat(CaseFormat.UPPER_CAMEL)
public boolean knowsJavascript;
}
```
Will produce a schema with fields "user", "age_in_years", and "KnowsJavascript".
I think the primary application of this will be using `lower_underscore` for schemas that will be used across languages (e.g. in LogicalType base types, and cross-language transform configuration objects), but I figured a general purpose solution based on [`CaseFormat`](https://guava.dev/releases/23.0/api/docs/com/google/common/base/CaseFormat.html) would be more valuable.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #13003:
URL: https://github.com/apache/beam/pull/13003#issuecomment-702904376
R: @kennknowles
CC: @reuvenlax @lukecwik
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #13003:
URL: https://github.com/apache/beam/pull/13003#discussion_r499734737
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
##########
@@ -118,6 +123,29 @@ public static FieldValueTypeInformation forField(Field field) {
.build();
}
+ public static <T extends AnnotatedElement & Member> String getNameOverride(
+ String original, T member) {
+ SchemaFieldName fieldName = member.getAnnotation(SchemaFieldName.class);
+ SchemaCaseFormat caseFormatAnnotation = member.getAnnotation(SchemaCaseFormat.class);
+ SchemaCaseFormat classCaseFormatAnnotation =
+ member.getDeclaringClass().getAnnotation(SchemaCaseFormat.class);
Review comment:
Worst cast, the person authoring `MyBeamObject` has to write a bunch of trivial overrides. I'm OK with that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #13003:
URL: https://github.com/apache/beam/pull/13003#issuecomment-727013754
Run Java_Examples_Dataflow PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #13003:
URL: https://github.com/apache/beam/pull/13003#issuecomment-726897629
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #13003:
URL: https://github.com/apache/beam/pull/13003#discussion_r499022266
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
##########
@@ -118,6 +123,29 @@ public static FieldValueTypeInformation forField(Field field) {
.build();
}
+ public static <T extends AnnotatedElement & Member> String getNameOverride(
+ String original, T member) {
+ SchemaFieldName fieldName = member.getAnnotation(SchemaFieldName.class);
+ SchemaCaseFormat caseFormatAnnotation = member.getAnnotation(SchemaCaseFormat.class);
+ SchemaCaseFormat classCaseFormatAnnotation =
+ member.getDeclaringClass().getAnnotation(SchemaCaseFormat.class);
Review comment:
Hm I didn't consider inheritance at all. I think your assessment about how this will behave is correct. I think this behavior would be fine, except in a context like this:
```java
class SomethingOutOfMyControl {
... fields ...
}
@SchemaCaseFormat(B)
class MyBeamObject extends SomethingOutOfMyControl { }
```
Not sure how often something like that comes up, but maybe that's enough reason to scope the case format to the class we're inferring the schema from.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #13003:
URL: https://github.com/apache/beam/pull/13003#discussion_r499005140
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
##########
@@ -118,6 +123,29 @@ public static FieldValueTypeInformation forField(Field field) {
.build();
}
+ public static <T extends AnnotatedElement & Member> String getNameOverride(
+ String original, T member) {
+ SchemaFieldName fieldName = member.getAnnotation(SchemaFieldName.class);
+ SchemaCaseFormat caseFormatAnnotation = member.getAnnotation(SchemaCaseFormat.class);
+ SchemaCaseFormat classCaseFormatAnnotation =
+ member.getDeclaringClass().getAnnotation(SchemaCaseFormat.class);
Review comment:
I hate inheritance but I have to ask: what happens with
```java
@SchemaCaseFormat(A)
class Parent {
... fields ...
}
@SchemaCaseFormat(B)
class Child extends Parent { }
```
I'm honestly not sure what should happen. I think in your case anything not overridden will get case format A. That seems right but just want to highlight it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #13003:
URL: https://github.com/apache/beam/pull/13003#issuecomment-727013655
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #13003:
URL: https://github.com/apache/beam/pull/13003
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a change in pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation
Posted by GitBox <gi...@apache.org>.
kennknowles commented on a change in pull request #13003:
URL: https://github.com/apache/beam/pull/13003#discussion_r499005140
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
##########
@@ -118,6 +123,29 @@ public static FieldValueTypeInformation forField(Field field) {
.build();
}
+ public static <T extends AnnotatedElement & Member> String getNameOverride(
+ String original, T member) {
+ SchemaFieldName fieldName = member.getAnnotation(SchemaFieldName.class);
+ SchemaCaseFormat caseFormatAnnotation = member.getAnnotation(SchemaCaseFormat.class);
+ SchemaCaseFormat classCaseFormatAnnotation =
+ member.getDeclaringClass().getAnnotation(SchemaCaseFormat.class);
Review comment:
I hate inheritance but I have to ask: what happens with
```
@SchemaCaseFormat(A)
class Parent {
... fields ...
}
@SchemaCaseFormat(B)
class Child extends Parent { }
```
I'm honestly not sure what should happen. I think in your case anything not overridden will get case format A. That seems right but just want to highlight it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org