You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Germain Tanguy <ta...@gmail.com> on 2012/10/13 01:25:29 UTC
Interference between two customSerde.jar
Hi everybody,
I am using Hive-0.9.0 and Hadoop-0.20.2. I have two customizes SerDe : custoSerde1.jar and custoSerde2.jar adding with ~/.hiverc .
In HDFS I have : /input/data_for_custoSerde1/
/input/data_for_custoSerde2/
When I run a query on my tableSerde1 it's work but not on tableSerde2. If I remove custoSerde1.jar of hiverc the query works on tableSerde2 and when I put custoSerde1.jar back in hiverc the query don't work anymore on tableSerde1 but it's continuing to work on tableSerde2. And inversely..
Does anyone have an idea why?
Is there any possibility to specify which SerDe.jar we want to use when we run a query?
All help will be welcome :)
Thanks and regards,
Germain.
Re: Interference between two customSerde.jar
Posted by Germain Tanguy <ta...@gmail.com>.
Thanks Roberto for you answer.
It turns out that the class names for the two serde were the sames... So I just had to change the name and it works.
By the way thank you for your tutorial on your blog, it helped me a lot to write my own serde.
Germain.
Le 12 oct. 2012 à 16:39, Roberto Congiu a écrit :
> Without more specific info about the two serdes I can't be sure, but most likely there may be a dependency conflict between the two serdes. Sometimes, for instance, they may use a different version of log4j, and if packaged with all dependencies they can indeed interfere with each other.
> Have a look at their POM, or jar -tf the serde to see if they have two different versions of the same class.
>
> R.
>
> On Fri, Oct 12, 2012 at 4:25 PM, Germain Tanguy <ta...@gmail.com> wrote:
> Hi everybody,
>
> I am using Hive-0.9.0 and Hadoop-0.20.2. I have two customizes SerDe : custoSerde1.jar and custoSerde2.jar adding with ~/.hiverc .
> In HDFS I have : /input/data_for_custoSerde1/
> /input/data_for_custoSerde2/
>
> When I run a query on my tableSerde1 it's work but not on tableSerde2. If I remove custoSerde1.jar of hiverc the query works on tableSerde2 and when I put custoSerde1.jar back in hiverc the query don't work anymore on tableSerde1 but it's continuing to work on tableSerde2. And inversely..
>
> Does anyone have an idea why?
> Is there any possibility to specify which SerDe.jar we want to use when we run a query?
>
> All help will be welcome :)
>
> Thanks and regards,
>
> Germain.
>
>
>
>
> --
> ----------------------------------------------------------
> Good judgement comes with experience.
> Experience comes with bad judgement.
> ----------------------------------------------------------
> Roberto Congiu - Data Engineer - OpenX
> tel: +1 626 466 1141
>
Re: Interference between two customSerde.jar
Posted by Roberto Congiu <ro...@openx.com>.
Without more specific info about the two serdes I can't be sure, but most
likely there may be a dependency conflict between the two serdes.
Sometimes, for instance, they may use a different version of log4j, and if
packaged with all dependencies they can indeed interfere with each other.
Have a look at their POM, or jar -tf the serde to see if they have two
different versions of the same class.
R.
On Fri, Oct 12, 2012 at 4:25 PM, Germain Tanguy <ta...@gmail.com>wrote:
> Hi everybody,
>
> I am using Hive-0.9.0 and Hadoop-0.20.2. I have two customizes SerDe :
> custoSerde1.jar and custoSerde2.jar adding with ~/.hiverc .
> In HDFS I have : /input/data_for_custoSerde1/
> /input/data_for_custoSerde2/
>
> When I run a query on my tableSerde1 it's work but not on tableSerde2. If
> I remove custoSerde1.jar of hiverc the query works on tableSerde2 and when
> I put custoSerde1.jar back in hiverc the query don't work anymore on
> tableSerde1 but it's continuing to work on tableSerde2. And inversely..
>
> Does anyone have an idea why?
> Is there any possibility to specify which SerDe.jar we want to use when we
> run a query?
>
> All help will be welcome :)
>
> Thanks and regards,
>
> Germain.
>
>
--
----------------------------------------------------------
Good judgement comes with experience.
Experience comes with bad judgement.
----------------------------------------------------------
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141