You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Kaymak, Tobias" <to...@ricardo.ch> on 2022/01/10 20:19:44 UTC

Kafka -> BigQueryIO Beam/Dataflow job ends up with weird encoding

Hello and Happy New Year!

I am migrating a Java Beam pipeline from 2.27.0 to 2.34.0 and from Flink to
Dataflow.

I have unit tests for the easy ParDo transforms but along the way somehow
my encoding gets screwed up. I replaced my JSON to TableRow step with the
one from the official Google/Teleport repo - with no effect:

    // Parse the JSON into a {@link TableRow} object.
    try (InputStream inputStream =
        new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))) {
      row = TableRowJsonCoder.of().decode(inputStream, Context.OUTER);

    } catch (IOException e) {
      throw new RuntimeException("Failed to serialize json to table row: "
+ json, e);
    }

The pipeline does nothing fancy except being able to read from ProtoBinary
and ProtoJson depending on a config.

My problematic result in BigQuery looks like this:

Z��rich

(should be Zürich).

Has anyone ever encountered this problem? Does it ring a bell?

Best,
Tobias

Re: Kafka -> BigQueryIO Beam/Dataflow job ends up with weird encoding

Posted by "Kaymak, Tobias" <to...@ricardo.ch>.
Hello Tobias,

Have you thought about the encoding of the String? No you have not - adding:

String input = new String(data, StandardCharsets.UTF_8);

Solved the issue for me.

Have a great 2022!

On Mon, Jan 10, 2022 at 9:19 PM Kaymak, Tobias <to...@ricardo.ch>
wrote:

> Hello and Happy New Year!
>
> I am migrating a Java Beam pipeline from 2.27.0 to 2.34.0 and from Flink
> to Dataflow.
>
> I have unit tests for the easy ParDo transforms but along the way somehow
> my encoding gets screwed up. I replaced my JSON to TableRow step with the
> one from the official Google/Teleport repo - with no effect:
>
>     // Parse the JSON into a {@link TableRow} object.
>     try (InputStream inputStream =
>         new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))) {
>       row = TableRowJsonCoder.of().decode(inputStream, Context.OUTER);
>
>     } catch (IOException e) {
>       throw new RuntimeException("Failed to serialize json to table row: "
> + json, e);
>     }
>
> The pipeline does nothing fancy except being able to read from ProtoBinary
> and ProtoJson depending on a config.
>
> My problematic result in BigQuery looks like this:
>
> Z��rich
>
> (should be Zürich).
>
> Has anyone ever encountered this problem? Does it ring a bell?
>
> Best,
> Tobias
>