Re: simultaneous read + write?

[Scott Carey <sc...@apache.org>]

AvroDataFile deals with this for some cases.  Is it an acceptable API for
your use case?  You can configure the block size to be very small and/or
flush() regularly.

If you do this on your own, you will need to track the position that you
start to read a record at, and if there is a failure, rewind and reset the
reader to that position.

-Scott


On 8/25/11 7:17 PM, "Yang" <te...@gmail.com> wrote:

>I'm trying to implement an on-disk queue, which contains avro records,
>SpecificRecord
>
>my queue implementation basically contains a
>SpecificDatumWriter, and a SpecificDatumReader  pointing to the same file
>.
>
>the problem is, that when the reader reaches the EOF, I can no longer
>use it again,
>even after I append more records to the file,  if I call the same
>SpecificDatumReader.read() again,
>it gave me exceptions:
>
>
>--------------------------------------------------------------------------
>-----
>Test set: blah.MyTest
>--------------------------------------------------------------------------
>-----
>Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257
>sec <<< FAILURE!
>testBasic(blah.MyTest)  Time elapsed: 0.24 sec  <<< ERROR!
>java.lang.ArrayIndexOutOfBoundsException
>    at java.lang.System.arraycopy(Native Method)
>    at 
>org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.j
>ava:670)
>    at 
>org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:453)
>    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120)
>    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405)
>    at 
>org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
>    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
>    at 
>org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
>    at 
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14
>2)
>    at 
>org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j
>ava:166)
>    at 
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13
>8)
>    at 
>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:12
>9)
>    at blah.DiskEventsQueue.dequeue2(MyTest.java:55)
>    at blah.MyTest.testBasic(MyTest.java:85)
>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    at 
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
>    at 
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
>    at java.lang.reflect.Method.invoke(Method.java:616)
>
>
>
>
>Thanks
>Yang

Re: simultaneous read + write?

[Yang <te...@gmail.com>]

thanks Scott, this is actually due to the bug that I filed. once that is
solved , it should be fine.

I created my own because I want to swap the encoder easily , right now
DataFile uses binaryEncoder only.  I do keep a separate marker which is
persisted *kind of* synchronously as I read out records.



On Fri, Sep 2, 2011 at 5:58 PM, Scott Carey <sc...@apache.org> wrote:

> AvroDataFile deals with this for some cases.  Is it an acceptable API for
> your use case?  You can configure the block size to be very small and/or
> flush() regularly.
>
> If you do this on your own, you will need to track the position that you
> start to read a record at, and if there is a failure, rewind and reset the
> reader to that position.
>
> -Scott
>
>
> On 8/25/11 7:17 PM, "Yang" <te...@gmail.com> wrote:
>
> >I'm trying to implement an on-disk queue, which contains avro records,
> >SpecificRecord
> >
> >my queue implementation basically contains a
> >SpecificDatumWriter, and a SpecificDatumReader  pointing to the same file
> >.
> >
> >the problem is, that when the reader reaches the EOF, I can no longer
> >use it again,
> >even after I append more records to the file,  if I call the same
> >SpecificDatumReader.read() again,
> >it gave me exceptions:
> >
> >
> >--------------------------------------------------------------------------
> >-----
> >Test set: blah.MyTest
> >--------------------------------------------------------------------------
> >-----
> >Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.257
> >sec <<< FAILURE!
> >testBasic(blah.MyTest)  Time elapsed: 0.24 sec  <<< ERROR!
> >java.lang.ArrayIndexOutOfBoundsException
> >    at java.lang.System.arraycopy(Native Method)
> >    at
> >org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill(BinaryDecoder.j
> >ava:670)
> >    at
> >org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:453)
> >    at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:120)
> >    at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:405)
> >    at
> >org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229)
> >    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> >    at
> >org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206)
> >    at
> >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:14
> >2)
> >    at
> >org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.j
> >ava:166)
> >    at
> >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:13
> >8)
> >    at
> >org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:12
> >9)
> >    at blah.DiskEventsQueue.dequeue2(MyTest.java:55)
> >    at blah.MyTest.testBasic(MyTest.java:85)
> >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >    at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >57)
> >    at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:43)
> >    at java.lang.reflect.Method.invoke(Method.java:616)
> >
> >
> >
> >
> >Thanks
> >Yang
>
>
>