[jira] [Commented] (BEAM-14431) Handle nulls using SnowflakeIO

["Kenneth Knowles (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536786#comment-17536786 ] 

Kenneth Knowles commented on BEAM-14431:
----------------------------------------

[~chamikara] do you know the best contact for SnowflakeIO issues?

> Handle nulls using SnowflakeIO
> ------------------------------
>
>                 Key: BEAM-14431
>                 URL: https://issues.apache.org/jira/browse/BEAM-14431
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-snowflake
>    Affects Versions: 2.38.0
>            Reporter: Varun
>            Priority: P1
>              Labels: snowflake
>         Attachments: csv_mapper_data.png, snowflake_data.png
>
>
> When SnowflakeIO is used within python beam code using ReadFromSnowflake [1] ; the returned data isn't able to list the nulls from snowflake side.
> The nulls from snowflake becomes b'N' ; which is a wrong value and gives incorrect value for the column. In case of strings the actual value can also be 'N'. So it becomes tough to handle it in csv_mapper.
> To mitigate this we are using query instead of table and using ifnull / coleasce within the select query for String columns. For non-string columns we are converting back b'N' to null or empty as per requirement.
>  
> [1] : [https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)