You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2023/02/28 22:16:34 UTC

[GitHub] [beam] ahmedabu98 opened a new issue, #25669: [Bug]: Different orderings of SchemaAwareExternalTransform() kwargs may result in misplaced arguments

ahmedabu98 opened a new issue, #25669:
URL: https://github.com/apache/beam/issues/25669

   ### What happened?
   
   Was testing a SchemaTransform Python wrapper (#25521) and found that I had to have a right ordering of kwargs for the input arguments to reach the Java transform in the right fields. This is weird because the ordering of kwargs should have no impact.
   
   For example, where `self._table="my_project:my_dataset.xlang_table"`,
   
   the following works fine:
   ```
   external_storage_write = SchemaAwareExternalTransform(
       identifier=self.schematransform_config.identifier,
       expansion_service=self._expansion_service,
       createDisposition=self._create_disposition,
       writeDisposition=self._write_disposition,     #<---
       triggeringFrequencySeconds=self._triggering_frequency,
       useAtLeastOnceSemantics=self._use_at_least_once,
       table=self._table)                            #<---
   ```
   and I get a configuration object in Java transform that looks like this:
   ```
   BigQueryStorageWriteApiSchemaTransformConfiguration{
     table=my_project:my_dataset.xlang_table, 
     createDisposition=, 
     writeDisposition=, 
     triggeringFrequencySeconds=0, 
     useAtLeastOnceSemantics=false}
   ```
   
   However, if I change the kwargs to look like this (switch places of table and writeDisposition):
   ```
   external_storage_write = SchemaAwareExternalTransform(
       identifier=self.schematransform_config.identifier,
       expansion_service=self._expansion_service,
       createDisposition=self._create_disposition,
       table=self._table,                            #<---
       triggeringFrequencySeconds=self._triggering_frequency,
       useAtLeastOnceSemantics=self._use_at_least_once,
       writeDisposition=self._write_disposition)     #<---
   ```
   I get the following configuration object. Notice the value intended for `table` is now in the `writeDisposition` field.
   ```
   BigQueryStorageWriteApiSchemaTransformConfiguration{
     table=, 
     createDisposition=, 
     writeDisposition=my_project:my_dataset.xlang_table, 
     triggeringFrequencySeconds=0, 
     useAtLeastOnceSemantics=false
   ```
   
   
   ### Issue Priority
   
   Priority: 1 (data loss / total loss of function)
   
   ### Issue Components
   
   - [X] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj closed issue #25669: [Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and Java RowCoder decoding

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj closed issue #25669: [Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and  Java RowCoder decoding
URL: https://github.com/apache/beam/issues/25669


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on issue #25669: [Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and Java RowCoder decoding

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #25669:
URL: https://github.com/apache/beam/issues/25669#issuecomment-1484210381

   Update: we found that this problem was due to line 342 in the `Schema::sorted` method below, which copies the field encoding positions of an unsorted Schema over to the sorted Schema that is generated. This mismatch between encoding positions and actual field indices in the resulting sorted Schema leads to this issue. When RowCoder tries to encode/decode a field, it looks for the encoding position of that field. Here we see RowCoder using the encoding positions or a pre-sorted Schema to try decoding a Row with sorted Schema, leading to what looks like field misplacement.
   
   https://github.com/apache/beam/blob/3a1b64106657346f436bc9011d894ca90408a69e/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L333-L345
   
   
   Context: TypedSchemaTransformProvider uses a class to represent the configuration of the SchemaTransform. The configuration schema is inferred from its class using `AutoValueSchema`. Unfortunately, the ordering of fields in the generated configuration schema is not guaranteed, so as a workaround we are sorting the fields alphabetically.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #25669: [Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and Java RowCoder decoding

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #25669:
URL: https://github.com/apache/beam/issues/25669#issuecomment-1528145963

   Observed similar issues in #26480. The fix of #25685 appears to be generic to schema but the same still happens for SchemaIO wrapper (used by jdbc xlang).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] ahmedabu98 commented on issue #25669: [Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and Java RowCoder decoding

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #25669:
URL: https://github.com/apache/beam/issues/25669#issuecomment-1482965773

   I followed this [unittest](https://github.com/apache/beam/blob/ddae966f3346fbe247486324cbf8a8a532895316/sdks/python/apache_beam/transforms/external_test.py#L449) and tried encoding my configuration with SchemaTransformPayloadBuilder() and decoding with proto_utils.parse_Bytes(). It returned a configuration with fields in the right order.
   
   Not sure what the mismatch is between how Python and Java handle the payload.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] chamikaramj commented on issue #25669: [Bug]: Configuration row arguments may get misplaced between Python SchemaTransformPayload encoding and Java RowCoder decoding

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #25669:
URL: https://github.com/apache/beam/issues/25669#issuecomment-1533691634

   jdbc.py has a similar issue but that's not addressed here. transforms in jdbc.py needs to preserve ordering since it uses a second config object [1] along with (old) SchemaIO API. This config object that is encoded in the Python side and decoded in the Java side. So ordering of the the objects in the two sides have to match.
   
   One way to fix this will be to update jdbc.py to use schema-aware transforms (which will pick up the fix provided in this PR).
   
   [1] https://github.com/apache/beam/blob/e868c8da79a8cf727044cf805b2041fbdb8ff181/sdks/python/apache_beam/io/jdbc.py#L339
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org