You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2024/05/22 16:28:21 UTC

[PR] Default SchemaTransform configs to snake_case [beam]

ahmedabu98 opened a new pull request, #31374:
URL: https://github.com/apache/beam/pull/31374

   Bringing back the changes reverted in #31109
   
   More details in #31061 and #31353


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2135884381

   > Thanks for the insight @lostluck! From a cursory search I see [xlang/bigtableio/bigtableio.go](https://github.com/apache/beam/blob/e764fc9c17dbc575d4a16b5e6454f4d38ef752be/sdks/go/pkg/beam/io/xlang/bigtableio/bigtable.go#L102) is currently the only SchemaTransform use-case in the Go SDK, is that right?
   
   I have no idea. I do not keep track of cross language implementations and details about specific IOs. Your search would be as good as mine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2135549050

   R: @lostluck 
   
   > manual field name override of the given snake case name.
   
   let me know if the last commit does what you meant here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2140460592

   Thanks @Polber for catching those, can you PTAL again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2135524195

   Thanks for the insight @lostluck! From a cursory search I see [xlang/bigtableio/bigtableio.go](https://github.com/apache/beam/blob/e764fc9c17dbc575d4a16b5e6454f4d38ef752be/sdks/go/pkg/beam/io/xlang/bigtableio/bigtable.go#L102) is currently the only SchemaTransform use-case in the Go SDK, is that right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2130287268

   Fixes #31353


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2145918706

   @chamikaramj would this impact using the transform service cross-version?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2130283093

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2135406221

   Probably does affect Go, but IIRC it just means non idiomatic go field names in generated structs. The existing schema handling code should handle making sure the field is Exported (upper case first character) with a manual field name override of the given snake case name.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on code in PR #31374:
URL: https://github.com/apache/beam/pull/31374#discussion_r1618157995


##########
sdks/python/apache_beam/yaml/standard_io.yaml:
##########
@@ -30,14 +30,14 @@
     mappings:
       'ReadFromBigQuery':
         query: 'query'
-        table: 'tableSpec'
-        fields: 'selectedFields'
-        row_restriction: 'rowRestriction'
+        table: 'table_spec'
+        fields: 'selected_fields'
+        row_restriction: 'row_restriction'
       'WriteToBigQuery':
         table: 'table'
-        create_disposition: 'createDisposition'
-        write_disposition: 'writeDisposition'
-        error_handling: 'errorHandling'
+        create_disposition: 'create_disposition'
+        write_disposition: 'write_disposition'
+        error_handling: 'error_handling'
         # TODO(https://github.com/apache/beam/issues/30058): Required until autosharding support is fixed
         num_streams: 'numStreams'

Review Comment:
   ```suggestion
           num_streams: 'num_streams'
   ```



##########
sdks/python/apache_beam/yaml/standard_io.yaml:
##########
@@ -206,25 +206,25 @@
     mappings:
       'ReadFromJdbc':
         driver_class_name: 'driverClassName'

Review Comment:
   ```suggestion
           driver_class_name: 'driver_class_name'
   ```



##########
sdks/python/apache_beam/yaml/standard_io.yaml:
##########


Review Comment:
   Also need modifications in https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/standard_providers.yaml



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2130281990

   R: @robertwb 
   R: @chamikaramj 
   
   CC: @lostluck (if this affects Go at all)
   CC: @Polber (if any more changes are needed for YAML)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2145644700

   @ahmedabu98 LGTM, but there is an open PR #31455 that would require a couple changes as well if it gets in before this PR is merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2125394720

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #31374:
URL: https://github.com/apache/beam/pull/31374


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Default SchemaTransform configs to snake_case [beam]

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #31374:
URL: https://github.com/apache/beam/pull/31374#issuecomment-2146011311

   So transform service just wrap expansion services. So change of behavior should be similar to the change of behavior of the Java expansion service.
   
   Regarding, cross-version usage, I think what could break is use-cases that assume a schema from a old Beam version while trying to use the new Beam version for expansion. For example, hand-coded wrappers in customer's side. We are updating the affected Beam provided wrappers so I think the chances of someone running into issues due to this should be rare.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org