You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2010/03/05 22:05:27 UTC
[jira] Created: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Demux depends on conf/chukwa-collector-conf.xml
-----------------------------------------------
Key: CHUKWA-460
URL: https://issues.apache.org/jira/browse/CHUKWA-460
Project: Hadoop Chukwa
Issue Type: Bug
Components: Data Processors
Reporter: Eric Yang
Assignee: Eric Yang
We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-460:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to both TRUNK and 0.4. Thanks Ari.
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-460.patch
>
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842024#action_12842024 ]
Bill Graham commented on CHUKWA-460:
------------------------------------
Testing the collector it appears it does not have this dependency, but instead picks up the HDFS configs correctly from the hadoop configs. Which is good, albeit confusing.
So really this config should be moved to the demux configs for now, to clear up potential confusion over who reads that value. Or better yet, Eric's approach of removing the config entirely and fixing Demux to not use that value should be followed.
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-460:
-----------------------------
Status: Patch Available (was: Open)
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-460.patch
>
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848454#action_12848454 ]
Eric Yang commented on CHUKWA-460:
----------------------------------
The documentation did not show writer.hdfs.filesystem, was required for demux. This change actually conforms to what is currently available on document.
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-460.patch
>
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848438#action_12848438 ]
Ari Rabkin commented on CHUKWA-460:
-----------------------------------
+1 to patch. For trunk as well as 0.4 branch. Do we need to do a documentation update?
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-460.patch
>
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842020#action_12842020 ]
Eric Yang commented on CHUKWA-460:
----------------------------------
Demux process should not be using writer.hdfs.filesystem. The original implementation made a mistake to use "writer.hdfs.filesystem" as reference to hdfs. Instead, demux should pick up the proper hdfs url from HADOOP_CONF_DIR. I don't have any objection on the suggested workaround, but less configuration that we need to maintain, the better.
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-460:
-----------------------------
Attachment: CHUKWA-460.patch
This patch will remove the reference of writer.hdfs.filesystem from demux and archive, and use HADOOP_CONF_DIR instead.
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-460.patch
>
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-460) Demux depends on
conf/chukwa-collector-conf.xml
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850980#action_12850980 ]
Hudson commented on CHUKWA-460:
-------------------------------
Integrated in Chukwa-trunk #330 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/330/])
> Demux depends on conf/chukwa-collector-conf.xml
> -----------------------------------------------
>
> Key: CHUKWA-460
> URL: https://issues.apache.org/jira/browse/CHUKWA-460
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-460.patch
>
>
> We've packaged our own chukwa rpms to be able to deploy the collector, the agent and the data processor each separately. In doing so I realized that the data processor (i.e., Demux) has a dependency on conf/chukwa-collector-conf.xml as packaged in the distro, since that's the only place where where 'writer.hdfs.filesystem' is set.
> The workaround is straight forward enough (just add writer.hdfs.filesystem to chukwa-demux.xml), but I thought I'd point out the cross-dependency. Although it feels redundant, it seems the proper thing to do would be to include writer.hdfs.filesystem in both configs, since both processes have that dep. Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.