[jira] [Created] (AVRO-3768) Provide InputStream implementation that wraps an Iterator of Record

["Victor (Jira)" <ji...@apache.org>]

Victor created AVRO-3768:
----------------------------

             Summary: Provide InputStream implementation that wraps an Iterator of Record
                 Key: AVRO-3768
                 URL: https://issues.apache.org/jira/browse/AVRO-3768
             Project: Apache Avro
          Issue Type: New Feature
          Components: java
            Reporter: Victor


Hello,

I have some code that generates avro record (using GenericData.Record precisely) record per record (I used an Iterator<GenericData.Record> in practice, but we could imagine anything similar including an actual java.util.Stream or even an avro-provided interface), and I would like to serialized it to some external system (an http request in my particular case). So basically the data is generated as it is pulled.


Right now, the only option I found is to use a combination of java.io.PipedInputStream and PipedOutputStream, wrapping the later inside a DataFileWriter and then feeding the records to the writer (in a separate thread) so that the PipedInputStream can be read by anything else.

As you can see this is a bit cumbersome and a more straightforward approach would be welcomed. I tried to implement this myself but I admit I got lost in all the moving pieces and I couldn't find a simple way of doing that without implementing a lot of low-level stuff. So maybe I'm missing something or we could add something to avro :)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)