You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2009/12/23 07:45:29 UTC
[jira] Created: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Custom processor classes not detected unless added to chukwa-core jar
---------------------------------------------------------------------
Key: CHUKWA-440
URL: https://issues.apache.org/jira/browse/CHUKWA-440
Project: Hadoop Chukwa
Issue Type: Bug
Components: Data Processors
Reporter: Bill Graham
http://wiki.apache.org/hadoop/DemuxModification
After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
The only way I've been able to make this work is to do the following:
1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Jiaqi Tan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793954#action_12793954 ]
Jiaqi Tan commented on CHUKWA-440:
----------------------------------
I believe in general, if a Hadoop MapReduce program is going to use class files from multiple jars, then the additional jar files need to be loaded using DistributedCache, and the corresponding Path added to the classpath to force them to be copied to every slave node that will run the code.
The way to get the functionality you're suggesting, I believe, would be to modify the Demux class to take in a list of extra jars and load them using DistributedCache.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794123#action_12794123 ]
Jerome Boulon commented on CHUKWA-440:
--------------------------------------
It's an Hadoop problem. Hadoop is running the demux using several machines so the jar needs to be available on all data nodes.
The way to use more than one jar is to add the -libjars option to your hadoop command, you can try that for Demux.
Let me know if it's working or not.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Status: Open (was: Patch Available)
Tested on my cluster, the current patch is incomplete.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794262#action_12794262 ]
Jerome Boulon commented on CHUKWA-440:
--------------------------------------
So if it's not working for Chukwa we need to fix it and I want to add the flag at the command line level since I'm no longer using DemuxManager because of other issues on S3, if we fix it inside DemuxManager then everyone that is only using Demux will still have the issue.
I'll do some testing but I don't have time right now, so if someone can test the libjar flag with a standard hadoop job that will be a good start.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Attachment: CHUKWA-440-2.patch
Revisied patch. This is the same as previous patch. It was working except I wasn't packaging Chukwa correctly during my test run. Additional parsers could be place in HDFS://hostname:port/chukwa/demux/*.jar and demux will pick it up.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440-1.patch, CHUKWA-440-2.patch, CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794138#action_12794138 ]
Eric Yang commented on CHUKWA-440:
----------------------------------
As I recall, the demux has been changed from command line submission to java deamon process. Demux job is submitted through ToolRunner. Hence, it is most likely to be required to modify the Demux Manager code. I am not 100% sure. Interested party should try both.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794128#action_12794128 ]
Eric Yang commented on CHUKWA-440:
----------------------------------
Line 317 of DemuxManager.java could add the additional jar files. We probably should put some logic to construct libjars by searching for CHUKWA_HOME/lib/demux/*.jar.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Status: Patch Available (was: Open)
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794132#action_12794132 ]
Jerome Boulon commented on CHUKWA-440:
--------------------------------------
-1 - Again this should not done by manually adding jars.
There's a generic way to do that in hadoop world using the -libjars flag, there's no point of adding chukwa specific code here, the problem has already been solved in hadoop.
So instead of adding some chukwa specific code to add the jar, we should enhance the demux script to look for additional jars and use the -libjars flag.
if the -libjars is not working for demux that's another issue but since we are implementing Tools it should already be supported.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Attachment: CHUKWA-440-1.patch
Using DistributedCache to load additional jar files. The idea is to put add on jar file in hdfs /chukwa/demux, and they will get loaded automatically. However, I can't get it to work. Jiaqi, could you review what is missing?
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440-1.patch, CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reassigned CHUKWA-440:
--------------------------------
Assignee: Eric Yang
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Status: Patch Available (was: Open)
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440-1.patch, CHUKWA-440-2.patch, CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801929#action_12801929 ]
Ari Rabkin commented on CHUKWA-440:
-----------------------------------
+1 to latest patch.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440-1.patch, CHUKWA-440-2.patch, CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this, thanks Ari.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440-1.patch, CHUKWA-440-2.patch, CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Jiaqi Tan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794257#action_12794257 ]
Jiaqi Tan commented on CHUKWA-440:
----------------------------------
@Jerome: If you use libjars, does that automatically copy the extra libs to every node? I don't believe it does, because I had problems with that the last time I tried, and similarly to Bill, I got it to work only by having everything in the main Chukwa package.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-440) Custom processor classes not
detected unless added to chukwa-core jar
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850976#action_12850976 ]
Hudson commented on CHUKWA-440:
-------------------------------
Integrated in Chukwa-trunk #330 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/330/])
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Assignee: Eric Yang
> Attachments: CHUKWA-440-1.patch, CHUKWA-440-2.patch, CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-440) Custom processor classes not detected
unless added to chukwa-core jar
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-440:
-----------------------------
Attachment: CHUKWA-440.patch
Converted demux to use mapred.used.genericoptionsparser flag to enable -libjars abc.jar,xyz.jar.
This change enable the ability to pass in -libjars for standalone demux, and demux running through demux manager.
Place the parser jar file in CHUKWA_HOME/lib/demux.
Yahoo is shutdown over this weekend, hence I can't test this code on my cluster. Anyone that could help in testing this code is highly appreciated.
> Custom processor classes not detected unless added to chukwa-core jar
> ---------------------------------------------------------------------
>
> Key: CHUKWA-440
> URL: https://issues.apache.org/jira/browse/CHUKWA-440
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: Data Processors
> Reporter: Bill Graham
> Attachments: CHUKWA-440.patch
>
>
> http://wiki.apache.org/hadoop/DemuxModification
> After implementing a custom parser as shown in the wiki link above and mapping it to a data type in chukwa-demux.xml, there's no easy way to register the class so it gets included in the job that's submitted to the hadoop cluster. I've added my jar containing the class to the lib/ directory of the chukwa data processor install and verified that it's in the classapath, but it's not submitted with the Hadoop job. On the Hadoop cluster ClassNotFoundExceptions are thrown in my mapper.
> The only way I've been able to make this work is to do the following:
> 1. put my class in the package org.apache.hadoop.chukwa.extraction.demux.processor.mapper
> 2. then manually add that class to the chukwa-core-0.3.0.jar that is on my data processor
> Instead, the class should be detected from whatever jar it lives in that's in the demux class path, regardless of the package it lives in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.