[jira] [Commented] (BEAM-11690) Python SqlTransform fails with an unhelpful Java error when type inference uses the pythonsdk_any fallback

["Brian Hulette (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272417#comment-17272417 ] 

Brian Hulette commented on BEAM-11690:
--------------------------------------

There's a question about where this error should be caught. I think I had an offline conversation with [~robertwb] where I floated the idea of just rejecting non-portable coders in ExternalTransform, but he pointed out that it's possible the expanding SDK understands pythonsdk_any.

We could provide the most helpful error if we catch this on the Python SDK side, in sql.py. I wouldn't want to be too aggressive with type checking there, as we could end up having two sources of truth for what types are supported by SQL (one in Java and one in Python). It's probably fine to just detect pythonsdk_any though. I think it would be appropriate to reject that for any schema-aware external transform that's using a jar expansion service.

> Python SqlTransform fails with an unhelpful Java error when type inference uses the pythonsdk_any fallback
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-11690
>                 URL: https://issues.apache.org/jira/browse/BEAM-11690
>             Project: Beam
>          Issue Type: Bug
>          Components: cross-language, sdk-py-core
>            Reporter: Brian Hulette
>            Assignee: Brian Hulette
>            Priority: P2
>
> See https://stackoverflow.com/questions/65903919/local-testing-apache-beam-with-sqltransform-in-python-sdk-i-receive-an-error for a good example of this.
> We should throw a more helpful error in this situation:
> - It should mention the problematic field(s)
> - It should suggest a solution (try wrapping in casts if using beam.Row, don't use Any in a NamedTuple)
> - It might point to some documentation about supported SQL types.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)