MemPipeline doesn't support writing Avro records via reflection.

[David Whiting <da...@gmail.com>]

It seems that this was half implemented but never finished, but I needed it
to create some test data to run through a cluster MapReduce test. The
current implementation correctly reflects the schema, but then uses a
GenericDatumWriter to try and write the record, causing a
ClassCastException. The correct way would be to get a ReflectDatumWriter
from the ReflectDataFactory.

I've attached a tiny patch that should fix it.

Re: MemPipeline doesn't support writing Avro records via reflection.

[Gabriel Reid <ga...@gmail.com>]

Hi David,

On Wed, Nov 20, 2013 at 4:00 PM, David Whiting <da...@gmail.com> wrote:
> It seems that this was half implemented but never finished, but I needed it
> to create some test data to run through a cluster MapReduce test. The
> current implementation correctly reflects the schema, but then uses a
> GenericDatumWriter to try and write the record, causing a
> ClassCastException. The correct way would be to get a ReflectDatumWriter
> from the ReflectDataFactory.
>
> I've attached a tiny patch that should fix it.

Yes, I think there are definitely some issues with the MemPipeline and
Avro handling.

It looks like your patch didn't make through with your mail (I think
all attachments get stripped out by the mailing list). Could you open
a JIRA ticket at https://issues.apache.org/jira/browse/CRUNCH and post
the patch there?

Thanks,

Gabriel