[jira] [Commented] (BEAM-10592) HDFS file writing has intermittent failures from 2.16 (and above)

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204141#comment-17204141 ] 

Beam JIRA Bot commented on BEAM-10592:
--------------------------------------

This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3.

Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean.


> HDFS file writing has intermittent failures from 2.16 (and above)
> -----------------------------------------------------------------
>
>                 Key: BEAM-10592
>                 URL: https://issues.apache.org/jira/browse/BEAM-10592
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-hadoop-file-system
>    Affects Versions: 2.16.0, 2.18.0, 2.19.0, 2.22.0
>         Environment: Azure Databricks
>            Reporter: Bharath
>            Priority: P2
>              Labels: stale-P2
>
> Bumping from Beam 2.13.0 to 2.16.0 and above we see broken pipelines running on spark/HDFS.
> Platform: Azure databricks.
> Beam 2.13.0 works fine, Have issues only after migrating to 2.16 and above, and only on large jobs (smaller jobs run fine)
> {{Caused by: java.io.IOException: Unable to rename resource wasbs://*****/output/npstand75k0727_1/np/.temp-beam-64a00562-5dcd-4bcd-9c5a-be7cff1231f3/483d5498-ed9c-46fd-b1ce-8647fa5c8a06 to wasbs://*******/output/npstand75k0727_1/np/confinements/part-00000-of-00001.txt. No further information provided by underlying filesystem.Caused by: java.io.IOException: Unable to rename resource wasbs://****/output/npstand75k0727_1/np/.temp-beam-64a00562-5dcd-4bcd-9c5a-be7cff1231f3/483d5498-ed9c-46fd-b1ce-8647fa5c8a06 to wasbs://****/output/npstand75k0727_1/np/confinements/part-00000-of-00001.txt. No further information provided by underlying filesystem. at org.apache.beam.sdk.io.hdfs.HadoopFileSystem.rename(HadoopFileSystem.java:287) at org.apache.beam.sdk.io.FileSystems.rename(FileSystems.java:327) at org.apache.beam.sdk.io.FileBasedSink$WriteOperation.moveToOutputFiles(FileBasedSink.java:755) at org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn.process(WriteFiles.java:850)}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)