You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Koert Kuipers <ko...@tresata.com> on 2013/10/21 17:41:47 UTC
expect specific record but get generic
i am observing that on a particular system (spark) my code breaks in that
avro does not return the specific record i expected but instead returns
generic records.
i suspect this is some class loading issue on the distributed system
(something about how the classpath is constructed for the spark slaves).
anyone had class loading issues get in the way of avro specific?
RE: expect specific record but get generic
Posted by 徐凡 <ti...@hotmail.com>.
I have the same issue. I changed all specific to generic, works fine now.
Date: Wed, 12 Feb 2014 08:39:10 -0500
Subject: Re: expect specific record but get generic
From: koert@tresata.com
To: user@avro.apache.org
Never solved it. Resorted to using generic for now.
On Feb 9, 2014 10:58 PM, "Andy He" <ho...@gmail.com> wrote:
Hi,
I am facing the exact problem when using Parquet+Avro in Spark? Did you have
the solution?
Thanks
--
View this message in context: http://apache-avro.679487.n3.nabble.com/expect-specific-record-but-get-generic-tp4028438p4029366.html
Sent from the Avro - Users mailing list archive at Nabble.com.
Re: expect specific record but get generic
Posted by Koert Kuipers <ko...@tresata.com>.
Never solved it. Resorted to using generic for now.
On Feb 9, 2014 10:58 PM, "Andy He" <ho...@gmail.com> wrote:
> Hi,
>
> I am facing the exact problem when using Parquet+Avro in Spark? Did you
> have
> the solution?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-avro.679487.n3.nabble.com/expect-specific-record-but-get-generic-tp4028438p4029366.html
> Sent from the Avro - Users mailing list archive at Nabble.com.
>
Re: expect specific record but get generic
Posted by Andy He <ho...@gmail.com>.
Hi,
I am facing the exact problem when using Parquet+Avro in Spark? Did you have
the solution?
Thanks
--
View this message in context: http://apache-avro.679487.n3.nabble.com/expect-specific-record-but-get-generic-tp4028438p4029366.html
Sent from the Avro - Users mailing list archive at Nabble.com.
Re: expect specific record but get generic
Posted by Koert Kuipers <ko...@tresata.com>.
thanks i will look into that
On Mon, Oct 21, 2013 at 4:49 PM, Doug Cutting <cu...@apache.org> wrote:
> On Mon, Oct 21, 2013 at 1:19 PM, Koert Kuipers <ko...@tresata.com> wrote:
> > doug, could it be a classloader (instead of classpath) issue? looking at
> > spark it seems to run the tasks inside the slaves/workers with a custom
> > classloader.
>
> Yes, it could be a classloader issue. Perhaps you need to pass the
> Spark task's ClassLoader to SpecificData? Find where your
> SpecificDatumReader is constructed, and pass it a SpecificData that
> was constructed with that ClassLoader.
>
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/specific/SpecificData.html#SpecificData(java.lang.ClassLoader)
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/specific/SpecificDatumReader.html#SpecificDatumReader(org.apache.avro.specific.SpecificData)
>
> Doug
>
Re: expect specific record but get generic
Posted by Doug Cutting <cu...@apache.org>.
On Mon, Oct 21, 2013 at 1:19 PM, Koert Kuipers <ko...@tresata.com> wrote:
> doug, could it be a classloader (instead of classpath) issue? looking at
> spark it seems to run the tasks inside the slaves/workers with a custom
> classloader.
Yes, it could be a classloader issue. Perhaps you need to pass the
Spark task's ClassLoader to SpecificData? Find where your
SpecificDatumReader is constructed, and pass it a SpecificData that
was constructed with that ClassLoader.
http://avro.apache.org/docs/current/api/java/org/apache/avro/specific/SpecificData.html#SpecificData(java.lang.ClassLoader)
http://avro.apache.org/docs/current/api/java/org/apache/avro/specific/SpecificDatumReader.html#SpecificDatumReader(org.apache.avro.specific.SpecificData)
Doug
Re: expect specific record but get generic
Posted by Koert Kuipers <ko...@tresata.com>.
doug, could it be a classloader (instead of classpath) issue? looking at
spark it seems to run the tasks inside the slaves/workers with a custom
classloader.
thanks! koert
On Mon, Oct 21, 2013 at 1:07 PM, Doug Cutting <cu...@apache.org> wrote:
> If the generated classes are not on the classpath then the generic
> representation is used. So, yes, this sounds like a classpath
> problem.
>
> On Mon, Oct 21, 2013 at 8:41 AM, Koert Kuipers <ko...@tresata.com> wrote:
> > i am observing that on a particular system (spark) my code breaks in that
> > avro does not return the specific record i expected but instead returns
> > generic records.
> >
> >
> > i suspect this is some class loading issue on the distributed system
> > (something about how the classpath is constructed for the spark slaves).
> >
> > anyone had class loading issues get in the way of avro specific?
>
Re: expect specific record but get generic
Posted by Doug Cutting <cu...@apache.org>.
If the generated classes are not on the classpath then the generic
representation is used. So, yes, this sounds like a classpath
problem.
On Mon, Oct 21, 2013 at 8:41 AM, Koert Kuipers <ko...@tresata.com> wrote:
> i am observing that on a particular system (spark) my code breaks in that
> avro does not return the specific record i expected but instead returns
> generic records.
>
>
> i suspect this is some class loading issue on the distributed system
> (something about how the classpath is constructed for the spark slaves).
>
> anyone had class loading issues get in the way of avro specific?