You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Dominik Schöneweiß (Jira)" <ji...@apache.org> on 2022/03/22 12:19:00 UTC

[jira] [Comment Edited] (BEAM-13319) Zstandard codec for Avro

    [ https://issues.apache.org/jira/browse/BEAM-13319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509914#comment-17509914 ] 

Dominik Schöneweiß edited comment on BEAM-13319 at 3/22/22, 12:18 PM:
----------------------------------------------------------------------

[~kenn] Would it be possible to upgrade the apache avro package to 1.9 or higher? The current version 1.8 is missing support for zstandard.

[https://github.com/apache/avro/blob/branch-1.9/lang/java/avro/src/main/java/org/apache/avro/file/CodecFactory.java]
[https://github.com/apache/avro/blob/branch-1.9/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java]

 

Edit: Maybe I should have added that I tried to update it myself, but ran into problems with the auto generated tests.


was (Author: nomnom):
[~kenn] Would it be possible to upgrade the apache avro package to 1.9 or higher? The current version 1.8 is missing support for zstandard.

https://github.com/apache/avro/blob/branch-1.9/lang/java/avro/src/main/java/org/apache/avro/file/CodecFactory.java
https://github.com/apache/avro/blob/branch-1.9/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java

> Zstandard codec for Avro
> ------------------------
>
>                 Key: BEAM-13319
>                 URL: https://issues.apache.org/jira/browse/BEAM-13319
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Igor Artamonov
>            Priority: P2
>              Labels: stale-P2, starter
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Currently AvroSource supports only snappy, deflate, bzip2 and xz codecs. 
> Avro library allows to write files with Zstandard codec too. Also BEAM-6365 added support of the codec but only for Text files.
> Would be helpful to have Zstandard for Avro as well.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)