You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Matteo Martignon (Jira)" <ji...@apache.org> on 2021/05/19 17:49:00 UTC
[jira] [Commented] (BEAM-8423) Japanese characters encoding issue
[ https://issues.apache.org/jira/browse/BEAM-8423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347801#comment-17347801 ]
Matteo Martignon commented on BEAM-8423:
----------------------------------------
Same happens for specific Spanish characters specific spanish characters like 'ñ' or accents 'á','é'. Dataflow workers default charset is US-ASCII.
{{Method }}{{getBytes(StandardCharsets.UTF_8) does not actually returns byte array UTF-8 encoded.}}
{{Example:}}
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/d2b43a5a19a1484833ea13761e6843b5b7d3328f/src/main/java/com/google/cloud/teleport/templates/common/BigQueryConverters.java#L333
> Japanese characters encoding issue
> -----------------------------------
>
> Key: BEAM-8423
> URL: https://issues.apache.org/jira/browse/BEAM-8423
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.15.0
> Environment: dataflow
> Reporter: Jyoti Aditya
> Priority: P3
>
> I am running apache beam job to parse japanese html pages. While running the job, I see in stackdriver log it is showing japanese character properly. But same data written to GCS bucket has encoding issue and it is getting corrupted.
>
> {noformat}
> //code
> Pipeline pipeline = Pipeline.create(options);
> CoderRegistry cr = pipeline.getCoderRegistry();
> cr.registerCoderForClass(String.class, StringUtf8Coder.of());
> cr.registerCoderForClass(Integer.class, BigEndianIntegerCoder.of());
> batchTuple = pipeline
> .apply("Read from input files", TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
> .apply("Process input files",ParDo.of(new ExtractDataFromHtmlPage(extractionConfig,beamConfig.getLoadingBucketURL())).withOutputTags(successRecord, TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)