You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Varun (Jira)" <ji...@apache.org> on 2022/05/06 07:51:00 UTC
[jira] [Updated] (BEAM-14431) Handle nulls using SnowflakeIO
[ https://issues.apache.org/jira/browse/BEAM-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Varun updated BEAM-14431:
-------------------------
Attachment: csv_mapper_data.png
snowflake_data.png
> Handle nulls using SnowflakeIO
> ------------------------------
>
> Key: BEAM-14431
> URL: https://issues.apache.org/jira/browse/BEAM-14431
> Project: Beam
> Issue Type: Bug
> Components: io-java-snowflake
> Affects Versions: 2.38.0
> Reporter: Varun
> Priority: P2
> Labels: snowflake
> Attachments: csv_mapper_data.png, snowflake_data.png
>
>
> When SnowflakeIO is used within python beam code using ReadFromSnowflake [1] ; the returned data isn't able to list the nulls from snowflake side.
> The nulls from snowflake becomes b'N' ; which is a wrong value and gives incorrect value for the column. In case of strings the actual value can also be 'N'. So it becomes tough to handle it in csv_mapper.
> To mitigate this we are using query instead of table and using ifnull / coleasce within the select query for String columns. For non-string columns we are converting back b'N' to null or empty as per requirement.
>
> [1] : [https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]|https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake]
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)