You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "kulkarni.swarnim@gmail.com" <ku...@gmail.com> on 2012/06/13 16:20:35 UTC
Providing a custom serialization.class SerDe property
Hello,
In order to provide a custom "serialization.class" to a SerDe, I created a
jar containing all my custom serialization classes and added them to the
hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
these custom classes via CLI, it still throws me a "ClassNotFoundException"
for those custom classes in my jar.
Is there something that I am missing? I confirmed that 'list jars' is
showing me the custom jar that I added.
Any help would be appreciated.
Thanks,
--
Swarnim
Re: Providing a custom serialization.class SerDe property
Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.
Another quick follow up on this itself.
If I try to run a query that invokes a M/R job, then it throws me this
exception:
java.io.FileNotFoundException: File does not exist: /Users/my-classes.jar
*at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:722)
*
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:77)
Looking at the stacktrace, it seems like it is searching for the jar on
HDFS rather than local filesystem. Is that intended? All "Select *" queries
that do not spawn a M/R job work fine.
Thanks,
On Wed, Jun 13, 2012 at 9:44 AM, kulkarni.swarnim@gmail.com <
kulkarni.swarnim@gmail.com> wrote:
> Cool. That worked!
>
> Thanks guys.
>
>
> On Wed, Jun 13, 2012 at 9:35 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Right, The end result is that same. The hive shell script currently
>> bulds a list of aux_lib jars and generates the auxpath arguments. but
>> it is good to know these files do not need to be in a static folder.
>>
>> On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
>> <BS...@stthomas.edu> wrote:
>> > Another option is to specify the path in the hive command:
>> >
>> > hive --auxpath ~/my-classes.jar
>> >
>> > -- Brad
>> >
>> > On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
>> >
>> >> You need to put these jars in your aux_lib folder or in your hadoop
>> >> classpath. There is a subtle difference between that classpath and the
>> >> classpath used by UDF and anything that involves a serde or input
>> >> format needs to be in auxlib.
>> >>
>> >> Edward
>> >>
>> >> On Wed, Jun 13, 2012 at 10:20 AM, kulkarni.swarnim@gmail.com
>> >> <ku...@gmail.com> wrote:
>> >>> Hello,
>> >>>
>> >>> In order to provide a custom "serialization.class" to a SerDe, I
>> created a
>> >>> jar containing all my custom serialization classes and added them to
>> the
>> >>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try
>> to use
>> >>> these custom classes via CLI, it still throws me a
>> "ClassNotFoundException"
>> >>> for those custom classes in my jar.
>> >>>
>> >>> Is there something that I am missing? I confirmed that 'list jars' is
>> >>> showing me the custom jar that I added.
>> >>>
>> >>> Any help would be appreciated.
>> >>>
>> >>> Thanks,
>> >>> --
>> >>> Swarnim
>> >
>>
>
>
>
> --
> Swarnim
>
--
Swarnim
Re: Providing a custom serialization.class SerDe property
Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.
Cool. That worked!
Thanks guys.
On Wed, Jun 13, 2012 at 9:35 AM, Edward Capriolo <ed...@gmail.com>wrote:
> Right, The end result is that same. The hive shell script currently
> bulds a list of aux_lib jars and generates the auxpath arguments. but
> it is good to know these files do not need to be in a static folder.
>
> On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
> <BS...@stthomas.edu> wrote:
> > Another option is to specify the path in the hive command:
> >
> > hive --auxpath ~/my-classes.jar
> >
> > -- Brad
> >
> > On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
> >
> >> You need to put these jars in your aux_lib folder or in your hadoop
> >> classpath. There is a subtle difference between that classpath and the
> >> classpath used by UDF and anything that involves a serde or input
> >> format needs to be in auxlib.
> >>
> >> Edward
> >>
> >> On Wed, Jun 13, 2012 at 10:20 AM, kulkarni.swarnim@gmail.com
> >> <ku...@gmail.com> wrote:
> >>> Hello,
> >>>
> >>> In order to provide a custom "serialization.class" to a SerDe, I
> created a
> >>> jar containing all my custom serialization classes and added them to
> the
> >>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try
> to use
> >>> these custom classes via CLI, it still throws me a
> "ClassNotFoundException"
> >>> for those custom classes in my jar.
> >>>
> >>> Is there something that I am missing? I confirmed that 'list jars' is
> >>> showing me the custom jar that I added.
> >>>
> >>> Any help would be appreciated.
> >>>
> >>> Thanks,
> >>> --
> >>> Swarnim
> >
>
--
Swarnim
Re: Providing a custom serialization.class SerDe property
Posted by Edward Capriolo <ed...@gmail.com>.
Right, The end result is that same. The hive shell script currently
bulds a list of aux_lib jars and generates the auxpath arguments. but
it is good to know these files do not need to be in a static folder.
On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
<BS...@stthomas.edu> wrote:
> Another option is to specify the path in the hive command:
>
> hive --auxpath ~/my-classes.jar
>
> -- Brad
>
> On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
>
>> You need to put these jars in your aux_lib folder or in your hadoop
>> classpath. There is a subtle difference between that classpath and the
>> classpath used by UDF and anything that involves a serde or input
>> format needs to be in auxlib.
>>
>> Edward
>>
>> On Wed, Jun 13, 2012 at 10:20 AM, kulkarni.swarnim@gmail.com
>> <ku...@gmail.com> wrote:
>>> Hello,
>>>
>>> In order to provide a custom "serialization.class" to a SerDe, I created a
>>> jar containing all my custom serialization classes and added them to the
>>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
>>> these custom classes via CLI, it still throws me a "ClassNotFoundException"
>>> for those custom classes in my jar.
>>>
>>> Is there something that I am missing? I confirmed that 'list jars' is
>>> showing me the custom jar that I added.
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks,
>>> --
>>> Swarnim
>
Re: Providing a custom serialization.class SerDe property
Posted by "Rubin, Bradley S." <BS...@stthomas.edu>.
Another option is to specify the path in the hive command:
hive --auxpath ~/my-classes.jar
-- Brad
On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
> You need to put these jars in your aux_lib folder or in your hadoop
> classpath. There is a subtle difference between that classpath and the
> classpath used by UDF and anything that involves a serde or input
> format needs to be in auxlib.
>
> Edward
>
> On Wed, Jun 13, 2012 at 10:20 AM, kulkarni.swarnim@gmail.com
> <ku...@gmail.com> wrote:
>> Hello,
>>
>> In order to provide a custom "serialization.class" to a SerDe, I created a
>> jar containing all my custom serialization classes and added them to the
>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
>> these custom classes via CLI, it still throws me a "ClassNotFoundException"
>> for those custom classes in my jar.
>>
>> Is there something that I am missing? I confirmed that 'list jars' is
>> showing me the custom jar that I added.
>>
>> Any help would be appreciated.
>>
>> Thanks,
>> --
>> Swarnim
Re: Providing a custom serialization.class SerDe property
Posted by Edward Capriolo <ed...@gmail.com>.
You need to put these jars in your aux_lib folder or in your hadoop
classpath. There is a subtle difference between that classpath and the
classpath used by UDF and anything that involves a serde or input
format needs to be in auxlib.
Edward
On Wed, Jun 13, 2012 at 10:20 AM, kulkarni.swarnim@gmail.com
<ku...@gmail.com> wrote:
> Hello,
>
> In order to provide a custom "serialization.class" to a SerDe, I created a
> jar containing all my custom serialization classes and added them to the
> hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
> these custom classes via CLI, it still throws me a "ClassNotFoundException"
> for those custom classes in my jar.
>
> Is there something that I am missing? I confirmed that 'list jars' is
> showing me the custom jar that I added.
>
> Any help would be appreciated.
>
> Thanks,
> --
> Swarnim