Demux depends on conf/chukwa-collector-conf.xml

[Bill Graham <bi...@gmail.com>]

Hi,

We've packaged our own chukwa rpms to be able to deploy the collector, the
agent and the data processor each separately. In doing so I realized that
the data processor (i.e., Demux) has a dependency on
conf/chukwa-collector-conf.xml as packaged in the distro, since that's the
only place where where 'writer.hdfs.filesystem' is set.

The workaround is straight forward enough (just add writer.hdfs.filesystem
to chukwa-demux.xml), but I thought I'd point out the cross-dependency.
Although it feels redundant, it seems the proper thing to do would be to
include writer.hdfs.filesystem in both configs, since both processes have
that dep. Thoughts?

thanks,
Bill

Re: Demux depends on conf/chukwa-collector-conf.xml

[Eric Yang <ey...@yahoo-inc.com>]

Demux process should not be using writer.hdfs.filesystem.  The original
implementation made a mistake to use ³writer.hdfs.filesystem² as reference
to hdfs.  Instead, demux should pick up the proper hdfs url from
HADOOP_CONF_DIR.  I don¹t have any objection on the suggested workaround,
but less configuration that we need to maintain, the better.

I file a bug for this issue:

https://issues.apache.org/jira/browse/CHUKWA-460

Regards,
Eric

On 3/5/10 12:34 PM, "Bill Graham" <bi...@gmail.com> wrote:

> Hi,
> 
> We've packaged our own chukwa rpms to be able to deploy the collector, the
> agent and the data processor each separately. In doing so I realized that the
> data processor (i.e., Demux) has a dependency on
> conf/chukwa-collector-conf.xml as packaged in the distro, since that's the
> only place where where 'writer.hdfs.filesystem' is set.
> 
> The workaround is straight forward enough (just add writer.hdfs.filesystem to
> chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although
> it feels redundant, it seems the proper thing to do would be to include
> writer.hdfs.filesystem in both configs, since both processes have that dep.
> Thoughts?
> 
> thanks,
> Bill
> 
>