You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Kaymak, Tobias" <to...@ricardo.ch> on 2022/01/10 20:19:44 UTC
Kafka -> BigQueryIO Beam/Dataflow job ends up with weird encoding
Hello and Happy New Year!
I am migrating a Java Beam pipeline from 2.27.0 to 2.34.0 and from Flink to
Dataflow.
I have unit tests for the easy ParDo transforms but along the way somehow
my encoding gets screwed up. I replaced my JSON to TableRow step with the
one from the official Google/Teleport repo - with no effect:
// Parse the JSON into a {@link TableRow} object.
try (InputStream inputStream =
new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))) {
row = TableRowJsonCoder.of().decode(inputStream, Context.OUTER);
} catch (IOException e) {
throw new RuntimeException("Failed to serialize json to table row: "
+ json, e);
}
The pipeline does nothing fancy except being able to read from ProtoBinary
and ProtoJson depending on a config.
My problematic result in BigQuery looks like this:
Z��rich
(should be Zürich).
Has anyone ever encountered this problem? Does it ring a bell?
Best,
Tobias
Re: Kafka -> BigQueryIO Beam/Dataflow job ends up with weird encoding
Posted by "Kaymak, Tobias" <to...@ricardo.ch>.
Hello Tobias,
Have you thought about the encoding of the String? No you have not - adding:
String input = new String(data, StandardCharsets.UTF_8);
Solved the issue for me.
Have a great 2022!
On Mon, Jan 10, 2022 at 9:19 PM Kaymak, Tobias <to...@ricardo.ch>
wrote:
> Hello and Happy New Year!
>
> I am migrating a Java Beam pipeline from 2.27.0 to 2.34.0 and from Flink
> to Dataflow.
>
> I have unit tests for the easy ParDo transforms but along the way somehow
> my encoding gets screwed up. I replaced my JSON to TableRow step with the
> one from the official Google/Teleport repo - with no effect:
>
> // Parse the JSON into a {@link TableRow} object.
> try (InputStream inputStream =
> new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))) {
> row = TableRowJsonCoder.of().decode(inputStream, Context.OUTER);
>
> } catch (IOException e) {
> throw new RuntimeException("Failed to serialize json to table row: "
> + json, e);
> }
>
> The pipeline does nothing fancy except being able to read from ProtoBinary
> and ProtoJson depending on a config.
>
> My problematic result in BigQuery looks like this:
>
> Z��rich
>
> (should be Zürich).
>
> Has anyone ever encountered this problem? Does it ring a bell?
>
> Best,
> Tobias
>