You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@avro.apache.org by "Martin Tzvetanov Grigorov (Jira)" <ji...@apache.org> on 2022/03/29 11:09:00 UTC

[jira] [Resolved] (AVRO-3472) Writing to an object store (without intermediate file)

     [ https://issues.apache.org/jira/browse/AVRO-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Tzvetanov Grigorov resolved AVRO-3472.
---------------------------------------------
    Resolution: Not A Problem

Please use the mailing lists for such kind of questions!

> Writing to an object store (without intermediate file)
> ------------------------------------------------------
>
>                 Key: AVRO-3472
>                 URL: https://issues.apache.org/jira/browse/AVRO-3472
>             Project: Apache Avro
>          Issue Type: New Feature
>          Components: java
>         Environment: OpenJDK version 11 on Ubuntu
>            Reporter: Kevin A Shaw
>            Priority: Minor
>              Labels: Java11
>
> We need to write a large number files for a production system to an Object Store (AWS S3 in this case) which cannot be represented by a java.io.File object.  We are currently using a DataFileWriter() to create a file in a /tmp directory, putting it to S3 and then deleting the file.  This is becoming cumbersome and just seems inefficient.  Is there a way to instead create the file using an in-memory object, instead of writing to disk?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)