[jira] [Work logged] (BEAM-13416) SqsIO.write input / construction of Sqs request

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-13416?focusedWorklogId=735227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-735227 ]

ASF GitHub Bot logged work on BEAM-13416:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 02/Mar/22 11:02
            Start Date: 02/Mar/22 11:02
    Worklog Time Spent: 10m 
      Work Description: mosche commented on pull request #16947:
URL: https://github.com/apache/beam/pull/16947#issuecomment-1056795786


   @reuvenlax I had a closer look why JavaFieldSchema does not work. There's quite some issues, and this is by far not an exhaustive list:
   
   - Only [public, non-static, non-transient fields](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java#L129-L151) are considered. Generated models in the AWS SDK are private, however. Using some kind of Field filter in JavaFieldSchema, that can then be overwritten, this could be addressed. But this obviously also means to generate different FieldValueGetter using ByteBuddy.
   - Next, more involved I suppose, such AWS SDK models contain final fields only, but there's no known creator. The annotation based approach isn't feasible for such "external" classes.
   - An annoyance, but has to be handled as well: These model classes may contain maps with non primitive types. This is not supported. Also, in some cases there might be recursive type hierarchies that can't be supported and have to be dealt with.
   
   Also, I've seen all the optimisations done in JavaFieldSchema and JavaBeanSchema. Honestly, at this point I don't bother much about that yet. I'm more interested to understand what the right approach forward is giving all the AWS IOs currently use hand crafted low level coders.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 735227)
    Time Spent: 1.5h  (was: 1h 20m)

> SqsIO.write input / construction of Sqs request
> -----------------------------------------------
>
>                 Key: BEAM-13416
>                 URL: https://issues.apache.org/jira/browse/BEAM-13416
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-aws
>            Reporter: Moritz Mack
>            Priority: P2
>              Labels: aws, aws-sdk-v2, sqs
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> SqsIO.write should support arbitrary types T and be implemented in terms of a mapper function that converts T into an AWS request.
> The current implementation of write as {{PTransform<PCollection<SendMessageRequest>, PDone>}} is a rather poor choice. It may requires an additional unnecessary serialization & deserialisation round. But even worse, the current {{SendMessageRequestCoder}} does not support any advanced configuration beyond queueUrl and messageBody and will silently drop it without even warning the user.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)