You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/05 00:31:57 UTC
[GitHub] [beam] damccorm opened a new issue, #21620: SnowflakeIO fails to recognize new-line char in a column
damccorm opened a new issue, #21620:
URL: https://github.com/apache/beam/issues/21620
When a snowflake string column contains new-line, the snowflakeIO fails to read it as a record. Instead two invalid records gets created.
I see the issue due to the usage of TextIO; as the exported extract from the snowflake contains the column data enclosed with quoted string, making it a valid CSV.
Possibly be solved if FileIO(with custom splitable dofn to handle big file) is used instead of TextIO.
Imported from Jira [BEAM-14526](https://issues.apache.org/jira/browse/BEAM-14526). Original Jira may contain additional context.
Reported by: Varun Rauthan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] sfc-gh-kbregula commented on issue #21620: SnowflakeIO fails to recognize new-line char in a column
Posted by GitBox <gi...@apache.org>.
sfc-gh-kbregula commented on issue #21620:
URL: https://github.com/apache/beam/issues/21620#issuecomment-1151483850
https://github.com/apache/beam/pull/17403 this should help, as we won't have a CSV file that is inherently prone to various special character problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #21620: SnowflakeIO fails to recognize new-line char in a column
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #21620:
URL: https://github.com/apache/beam/issues/21620#issuecomment-1151430533
@sfc-gh-kbregula any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org