You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Jyoti Aditya (Jira)" <ji...@apache.org> on 2019/10/17 12:54:00 UTC
[jira] [Created] (BEAM-8423) Japanese characters encoding issue
Jyoti Aditya created BEAM-8423:
----------------------------------
Summary: Japanese characters encoding issue
Key: BEAM-8423
URL: https://issues.apache.org/jira/browse/BEAM-8423
Project: Beam
Issue Type: Bug
Components: beam-model
Affects Versions: 2.15.0
Environment: dataflow
Reporter: Jyoti Aditya
I am running apache beam job to parse japanese html pages. While running the job, I see in stackdriver log it is showing japanese character properly. But same data written to GCS bucket has encoding issue and it is getting corrupted.
{noformat}
//code
Pipeline pipeline = Pipeline.create(options);
CoderRegistry cr = pipeline.getCoderRegistry();
cr.registerCoderForClass(String.class, StringUtf8Coder.of());
cr.registerCoderForClass(Integer.class, BigEndianIntegerCoder.of());
batchTuple = pipeline
.apply("Read from input files",
TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
.apply("Process input files",
ParDo.of(new ExtractDataFromHtmlPage(extractionConfig, beamConfig.getLoadingBucketURL()))
.withOutputTags(successRecord, TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)