[jira] [Commented] (AVRO-3768) Provide InputStream implementation that wraps an Iterator of Record

["Christophe Le Saec (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730571#comment-17730571 ] 

Christophe Le Saec commented on AVRO-3768:
------------------------------------------

Hello Victor,
As far as i understood, you want a kind of queue of avro records that can receive records; and, on trigger rule (_number of record; volume in octet, time_) serialize the received records and send it to external system (by http in your particular case :)).

Goal of Apache avro is to offer *structured data serialization system* (that's where you use  GenericData.Record, DataFileWriter ...).
But for queueing system (_PipedInputStream / PipedOutputStream is one example_), i think it's not the role of Avro to cover it, but may be i missed something in you issue.
Could you put some extract of your code to render it more clear ?

Regards,
Christophe.

> Provide InputStream implementation that wraps an Iterator of Record
> -------------------------------------------------------------------
>
>                 Key: AVRO-3768
>                 URL: https://issues.apache.org/jira/browse/AVRO-3768
>             Project: Apache Avro
>          Issue Type: New Feature
>          Components: java
>            Reporter: Victor
>            Priority: Major
>
> Hello,
> I have some code that generates avro record (using GenericData.Record precisely) record per record (I used an Iterator<GenericData.Record> in practice, but we could imagine anything similar including an actual java.util.Stream or even an avro-provided interface), and I would like to serialized it to some external system (an http request in my particular case). So basically the data is generated as it is pulled.
> Right now, the only option I found is to use a combination of java.io.PipedInputStream and PipedOutputStream, wrapping the later inside a DataFileWriter and then feeding the records to the writer (in a separate thread) so that the PipedInputStream can be read by anything else.
> As you can see this is a bit cumbersome and a more straightforward approach would be welcomed. I tried to implement this myself but I admit I got lost in all the moving pieces and I couldn't find a simple way of doing that without implementing a lot of low-level stuff. So maybe I'm missing something or we could add something to avro :)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)