[jira] [Updated] (BEAM-8423) Japanese characters encoding issue

["Kenneth Knowles (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-8423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-8423:
----------------------------------
    Status: Open  (was: Triage Needed)

> Japanese characters encoding issue 
> -----------------------------------
>
>                 Key: BEAM-8423
>                 URL: https://issues.apache.org/jira/browse/BEAM-8423
>             Project: Beam
>          Issue Type: Bug
>          Components: beam-model
>    Affects Versions: 2.15.0
>         Environment: dataflow
>            Reporter: Jyoti Aditya
>            Priority: Major
>
> I am running apache beam job to parse japanese html pages. While running the job, I see in stackdriver log it is showing japanese character properly. But same data written to GCS bucket has encoding issue and it is getting corrupted.
>  
> {noformat}
> //code
> Pipeline pipeline = Pipeline.create(options);
> CoderRegistry cr = pipeline.getCoderRegistry();
> cr.registerCoderForClass(String.class, StringUtf8Coder.of());
> cr.registerCoderForClass(Integer.class, BigEndianIntegerCoder.of());
> batchTuple = pipeline
> 		.apply("Read from input files", TextIO.read().from(options.getloadingBucketURL()).withCompression(Compression.GZIP)).setCoder(StringUtf8Coder.of())
> 		.apply("Process input files",ParDo.of(new ExtractDataFromHtmlPage(extractionConfig,beamConfig.getLoadingBucketURL())).withOutputTags(successRecord, TupleTagList.of(errorRecord).and(deadLetterRecords)));{noformat}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)