[GitHub] [beam] lostluck opened a new issue, #24339: [New Feature][Go SDK]: Support user use of the Beam Iterable Coder.

[GitBox <gi...@apache.org>]

lostluck opened a new issue, #24339:
URL: https://github.com/apache/beam/issues/24339

   ### What happened?
   
   There's no supported way for users to generate types that use an [iterable coder](https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L846) as the outer most coder type in the Go SDK.
   This prevents being able to use certain cross language transforms. In particular, the python Run Inference transform requires using a Beam Standard Iterable coder coded values as input. 
   
   There are essentially two options for resolving this.
   
   Option 1: Change slices to use iterable coder instead of the custom coder it currently uses. 
   The current custom coder adds an extra length prefix to the wrapping, and is also non-portable. Making this change would constitute a breaking change due to the change in coder semantics. However, it's unlikely to cause significant issues unless a pipeline has already started using the State API with slices. This change would also prevent pipeline updates across the versions if a slice coder is being used.
   
   Option 2: Revamp the coder system to enable better access to "standard" coders and similar, special casing them. This means users who wish to use the Iterable coder would need to learn/know/discover this.
   
   This author prefers option 1, because it's much cleaner longer term, and is more obvious a solution. Option 2 would take longer to develop, and impact a much larger surface area of the SDK. Further, option 2 would be less obvious and less natural to Go programmers, who use slices more casually.
   
   Option 1 also requires relaxing the automatic inference that a "KV<k, Iter<v>>" coder is a CoGBK<K, V> coder. This approach was used because previously the only source of Iterable coders was from a side input or the result of a CoGBK. This can be handled at execution time by validating the consumer of a datasource is expecting a CoGBK or a slice.
   
   Making this change arguably opens up using an iterable function instead of a slice for that concrete value (previously this was the other way around for side input values), but that would be a separate extension.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: sdk-go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse closed issue #24339: [New Feature][Go SDK]: Support user use of the Beam Iterable Coder.

[GitBox <gi...@apache.org>]

riteshghorse closed issue #24339: [New Feature][Go SDK]: Support user use of the Beam Iterable Coder.
URL: https://github.com/apache/beam/issues/24339


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org