[jira] [Work logged] (AVRO-3266) Output stream incompatible with MagicS3GuardCommitter

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-3266?focusedWorklogId=745771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745771 ]

ASF GitHub Bot logged work on AVRO-3266:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 22/Mar/22 12:12
            Start Date: 22/Mar/22 12:12
    Worklog Time Spent: 10m 
      Work Description: eejbyfeldt opened a new pull request #1618:
URL: https://github.com/apache/avro/pull/1618


   Title: AVRO-3266: Be compatible with hadoop 3.x PathOutputCommitter
   
   In hadoop 3.x the abstract class PathOutputCommitter defines the method `getWorkPath()`, but in hadoop 2.x it only defined on FileOutputCommitter. So to be compatible with both hadoop 2.x and 3.x and support committers that does only implements PathOutputComitter and not FileOutputCommitter we make the call to getWorkPath using reflection.
   
   The implementation is based on the suggestions in https://github.com/apache/avro/pull/1431 but at that PR has no updates from the author in 3 months I create a new PR.
   
   This fixes [AVRO-3266](https://issues.apache.org/jira/browse/AVRO-3266)
   
   ### Tests
   
   Is covered by existing tests.
   
   ### Documentation
   
   No new public methods.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 745771)
    Time Spent: 1h  (was: 50m)

> Output stream incompatible with MagicS3GuardCommitter
> -----------------------------------------------------
>
>                 Key: AVRO-3266
>                 URL: https://issues.apache.org/jira/browse/AVRO-3266
>             Project: Apache Avro
>          Issue Type: Bug
>          Components: java
>            Reporter: Michiel de Jong
>            Priority: Minor
>              Labels: pull-request-available
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> Avro's output stream can not be used in combination with the MagicS3GuardCommitter
> {code:java}
> Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter{code}
> The reason for this problem is that AvroOutputFormatBase.getAvroFileOutputStream tries to cast the outputcommiter to a FileOutputCommitter.
> It can be solved by casting to a PathOutputCommitter instead (which is a superclass of both the FileOutputCommitter and the MagicS3GuardCommitter)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)