[jira] [Commented] (AVRO-3266) Output stream incompatible with MagicS3GuardCommitter

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561565#comment-17561565 ] 

ASF subversion and git services commented on AVRO-3266:
-------------------------------------------------------

Commit 797cec66c4d4b2d4eaf909bfc2a7d98922e2b128 in avro's branch refs/heads/master from Emil Ejbyfeldt
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=797cec66c ]

AVRO-3266: Work with hadoop 3.x PathOutputComitters (#1618)

* AVRO-3266: Work with hadoop 3.x PathOutputComitters

In hadoop 3.x the abstract class PathOutputCommitter defines the method
`getWorkPath()`, but in hadoop 2.x it only defined on
FileOutputCommitter. So to be compatible with both hadoop 2.x and 3.x
and support committers that does only implements PathOutputComitter and
not FileOutputCommitter we make the call to getWorkPath using
reflection.

* AVRO-3266: Add committer class name to exception

* Add comment on better fix when hadoop2 is dropped

Co-authored-by: Ryan Skraba <ry...@skraba.com>

> Output stream incompatible with MagicS3GuardCommitter
> -----------------------------------------------------
>
>                 Key: AVRO-3266
>                 URL: https://issues.apache.org/jira/browse/AVRO-3266
>             Project: Apache Avro
>          Issue Type: Bug
>          Components: java
>            Reporter: Michiel de Jong
>            Assignee: Emil Ejbyfeldt
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 1.11.1
>
>          Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Avro's output stream can not be used in combination with the MagicS3GuardCommitter
> {code:java}
> Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter{code}
> The reason for this problem is that AvroOutputFormatBase.getAvroFileOutputStream tries to cast the outputcommiter to a FileOutputCommitter.
> It can be solved by casting to a PathOutputCommitter instead (which is a superclass of both the FileOutputCommitter and the MagicS3GuardCommitter)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)