Add user define jars

[Jerome Boulon <jb...@netflix.com>]

Hi,
I would like to get feedback from people using their own external jars with
Demux.
The current implementation is using the distributed cache to upload jars to
Hadoop.
- Is it working well?
- Do you have any problem with this feature?

I¹m asking this because we solved this requirement in a different way in
Honu and I wonder if that¹s something we need to improve in Chukwa.

Thanks in advance,
  /Jerome.

Re: Add user define jars

[Jerome Boulon <jb...@netflix.com>]

In Honu, similar to what we have in chukwa, we have a directory where users
can add their own jars (csv parser for example) and I append those jars to
chukwa-core.jar at build time.

So at the end I have:
Jar -tvf Chukwa-core.jar
org/apache/hadoop/chukwa/util/RecordConstants.class
org/apache/hadoop/chukwa/XXXX
lib/user1.jar
lib/user2.jar
lib/user3.jar

The advantage I can see is that there's only 1 jar but you can still
"unpack" everything and get a clean understanding of what is what.
Hadoop is able to load classes from those jar without doing anything special
(aka hadoop jar chukwa-core.jar XXXX is working as expected).

/Jerome.

On 5/11/10 5:32 PM, "Ariel Rabkin" <as...@gmail.com> wrote:

> Can you say what approach you took in Honu?
> 
> On Tue, May 11, 2010 at 5:29 PM, Jerome Boulon <jb...@netflix.com> wrote:
>> Hi,
>> I would like to get feedback from people using their own external jars with
>> Demux.
>> The current implementation is using the distributed cache to upload jars to
>> Hadoop.
>> - Is it working well?
>> - Do you have any problem with this feature?
>> 
>> I¹m asking this because we solved this requirement in a different way in
>> Honu and I wonder if that¹s something we need to improve in Chukwa.
>> 
>> Thanks in advance,
>>   /Jerome.
> 
> 

Re: Add user define jars

[Ariel Rabkin <as...@gmail.com>]

Can you say what approach you took in Honu?

On Tue, May 11, 2010 at 5:29 PM, Jerome Boulon <jb...@netflix.com> wrote:
> Hi,
> I would like to get feedback from people using their own external jars with
> Demux.
> The current implementation is using the distributed cache to upload jars to
> Hadoop.
> - Is it working well?
> - Do you have any problem with this feature?
>
> I’m asking this because we solved this requirement in a different way in
> Honu and I wonder if that’s something we need to improve in Chukwa.
>
> Thanks in advance,
>   /Jerome.



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Re: Add user define jars

[Jerome Boulon <jb...@netflix.com>]

Sorry for the late answer, I was at GoogleIO.
Here how I¹m adding jars to chukwa-core.jar:

Add this to the chukwa_core target:

<jar destfile="./chukwa-core.jar" update="true">
       <zipfileset file="../ext/jars/other.jar"
          fullpath="lib/other.jar"/>
</jar>


/Jerome.

On 5/19/10 10:16 PM, "Bill Graham" <bi...@gmail.com> wrote:

> Hi Jerome,
> 
> I haven't had to use external jars yet (I've pushed the logic I need into
> chukwa instead), but I would like to have a solution where including an
> external jar is just a matter of adding a jar to a local directory. Similar to
> how Hive has the auxlibs/ dir. It sounds like your solution aligns with this
> approach. If that's the case, I'm in favor of it.
> 
> thanks,
> Bill
> 
> On Wed, May 19, 2010 at 6:02 PM, Kirk True <ki...@mustardgrain.com> wrote:
>>  
>> Hi Jerome,
>> 
>> I'm trying to use the 'stick your JAR in HDFS' "chukwa/demux" directory'
>> approach. I'm not able to get it working (see the "Chukwa can't find Demux
>> class" thread in the mailing list).
>> 
>> Thanks,
>> Kirk
>> 
>> 
>> On 5/11/10 5:29 PM, Jerome Boulon wrote:
>>>   Hi,
>>> I would like to get feedback from people using their own external jars with
>>> Demux.
>>> The current implementation is using the distributed cache to upload jars to
>>> Hadoop.
>>> - Is it working well?
>>> - Do you have any problem with this feature?
>>>  
>>> I¹m asking this because we solved this requirement in a different way in
>>> Honu and I wonder if that¹s something we need to improve in Chukwa.
>>>  
>>> Thanks in advance,
>>>   /Jerome. 
> 
> 

Re: Add user define jars

[Bill Graham <bi...@gmail.com>]

Hi Jerome,

I haven't had to use external jars yet (I've pushed the logic I need into
chukwa instead), but I would like to have a solution where including an
external jar is just a matter of adding a jar to a local directory. Similar
to how Hive has the auxlibs/ dir. It sounds like your solution aligns with
this approach. If that's the case, I'm in favor of it.

thanks,
Bill

On Wed, May 19, 2010 at 6:02 PM, Kirk True <ki...@mustardgrain.com> wrote:

>  Hi Jerome,
>
> I'm trying to use the 'stick your JAR in HDFS' "chukwa/demux" directory'
> approach. I'm not able to get it working (see the "Chukwa can't find Demux
> class" thread in the mailing list).
>
> Thanks,
> Kirk
>
>
> On 5/11/10 5:29 PM, Jerome Boulon wrote:
>
> Hi,
> I would like to get feedback from people using their own external jars with
> Demux.
> The current implementation is using the distributed cache to upload jars to
> Hadoop.
> - Is it working well?
> - Do you have any problem with this feature?
>
> I’m asking this because we solved this requirement in a different way in
> Honu and I wonder if that’s something we need to improve in Chukwa.
>
> Thanks in advance,
>   /Jerome.
>
>

Re: Add user define jars

[Kirk True <ki...@mustardgrain.com>]

Hi Jerome,

I'm trying to use the 'stick your JAR in HDFS' "chukwa/demux" directory' 
approach. I'm not able to get it working (see the "Chukwa can't find 
Demux class" thread in the mailing list).

Thanks,
Kirk

On 5/11/10 5:29 PM, Jerome Boulon wrote:
> Hi,
> I would like to get feedback from people using their own external jars 
> with Demux.
> The current implementation is using the distributed cache to upload 
> jars to Hadoop.
> - Is it working well?
> - Do you have any problem with this feature?
>
> I'm asking this because we solved this requirement in a different way 
> in Honu and I wonder if that's something we need to improve in Chukwa.
>
> Thanks in advance,
>   /Jerome.