You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Facundo Tomatis <fa...@gmail.com> on 2024/11/26 22:30:05 UTC
[QUESTION] Support for different String encoding in RowCoder
Hello everyone!
I've been developing a csv connector that wraps CsvIO, the read
operation outputs PCollection<Row> and the write operation takes
PCollection<Row>. I am having issues setting the encoding of the
resulting file and the input file, for example I would like to write a
CSV with ISO-8859-1 encoding or windows-1250 and more, and read from
those encodings as well.
Reading the source code I found out that Row's String fields (generated
with RowCoder.of(schema)) have a StringUtf8Encoder associated, is there
a way to change this encoder to be a custom encoder while maintaining
PCollection<Row>?
Thanks for your time.
Facu.
Re: [QUESTION] Support for different String encoding in RowCoder
Posted by Damon Douglas <da...@apache.org>.
Hello Facu,
Thank you for bringing this up. We've had this GitHub issue: https://github.com/apache/beam/issues/32485 that tracks this very issue. I wrote a candidate solution to what might work to refactor CsvIOParse to achieve this feature.
Feel free to continue the communication on that issue, if you would like to do a pull request. I would be grateful to collaborate with you, if you have the time and energy.
Thank you, again,
Best,
Damon
On 2024/11/26 22:30:05 Facundo Tomatis wrote:
> Hello everyone!
>
> I've been developing a csv connector that wraps CsvIO, the read
> operation outputs PCollection<Row> and the write operation takes
> PCollection<Row>. I am having issues setting the encoding of the
> resulting file and the input file, for example I would like to write a
> CSV with ISO-8859-1 encoding or windows-1250 and more, and read from
> those encodings as well.
>
> Reading the source code I found out that Row's String fields (generated
> with RowCoder.of(schema)) have a StringUtf8Encoder associated, is there
> a way to change this encoder to be a custom encoder while maintaining
> PCollection<Row>?
>
> Thanks for your time.
>
> Facu.
>
>
Re: [QUESTION] Support for different String encoding in RowCoder
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
+1 to RowCoder being an implementation detail (and it doesn't make
much sense to parameterize its string encodings, logically it just has
string fields).
It does make sense, however, to augment CsvIO to be able to name an
encoding that is used to decode the bytes of the file (producing
"standard" Rows, which would presumably have in memory representations
as java Strings and the encoding chosen for them would not be an
issue).
On Wed, Nov 27, 2024 at 4:56 PM Reuven Lax via dev <de...@beam.apache.org> wrote:
>
> The RowCoder encoding is not really intended to be an external encoding - i.e. it's not intended to be a stable encoding for writing into files. While it's fine to take in PCollection<Row> in your write operation, I would not recommend just using RowCoder in order to generate the bytes written to the file.
>
> On Wed, Nov 27, 2024 at 1:46 PM Facundo Tomatis <fa...@gmail.com> wrote:
>>
>> Hello everyone!
>>
>> I've been developing a csv connector that wraps CsvIO, the read
>> operation outputs PCollection<Row> and the write operation takes
>> PCollection<Row>. I am having issues setting the encoding of the
>> resulting file and the input file, for example I would like to write a
>> CSV with ISO-8859-1 encoding or windows-1250 and more, and read from
>> those encodings as well.
>>
>> Reading the source code I found out that Row's String fields (generated
>> with RowCoder.of(schema)) have a StringUtf8Encoder associated, is there
>> a way to change this encoder to be a custom encoder while maintaining
>> PCollection<Row>?
>>
>> Thanks for your time.
>>
>> Facu.
>>
Re: [QUESTION] Support for different String encoding in RowCoder
Posted by Reuven Lax via dev <de...@beam.apache.org>.
The RowCoder encoding is not really intended to be an external encoding -
i.e. it's not intended to be a stable encoding for writing into files.
While it's fine to take in PCollection<Row> in your write operation, I
would not recommend just using RowCoder in order to generate the bytes
written to the file.
On Wed, Nov 27, 2024 at 1:46 PM Facundo Tomatis <fa...@gmail.com>
wrote:
> Hello everyone!
>
> I've been developing a csv connector that wraps CsvIO, the read
> operation outputs PCollection<Row> and the write operation takes
> PCollection<Row>. I am having issues setting the encoding of the
> resulting file and the input file, for example I would like to write a
> CSV with ISO-8859-1 encoding or windows-1250 and more, and read from
> those encodings as well.
>
> Reading the source code I found out that Row's String fields (generated
> with RowCoder.of(schema)) have a StringUtf8Encoder associated, is there
> a way to change this encoder to be a custom encoder while maintaining
> PCollection<Row>?
>
> Thanks for your time.
>
> Facu.
>
>