You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Kirk True (JIRA)" <ji...@apache.org> on 2010/05/20 17:13:53 UTC

[jira] Created: (CHUKWA-488) Hadoop cannot find custom Demux class

Hadoop cannot find custom Demux class
-------------------------------------

                 Key: CHUKWA-488
                 URL: https://issues.apache.org/jira/browse/CHUKWA-488
             Project: Hadoop Chukwa
          Issue Type: Bug
          Components: MR Data Processors
    Affects Versions: 0.4.0
         Environment: Linux x86-64
Java 1.6.0_20
            Reporter: Kirk True


I'm getting ClassNotFoundException errors when running inside Hadoop's map phase, unable to find my class org.apache.hadoop.chukwa.extraction.demux.processor.mapper.XmlBasedDemux which I've packaged in a JAR named data-collection-demux-0.1.jar.

The problem seems to be in the values of these two properties in the Hadoop job configuration:

{code}
<property>
    <name>mapred.job.classpath.files</name>
    <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
</property>
<property>
    <name>mapred.cache.files</name>
    <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
</property>
{code}

The problem seems to stem from the fact that the call to DistributedCache.addFileToClassPath is passing in a Path that is in URI form, i.e. hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar whereas the DistributedCache API expects it to be a filesystem-based path (i.e. /chukwa/demux/data-collection-demux-0.1.jar). I'm not sure why, but the FileStatus object returned by FileSystem.listStatus is returning a URL-based path instead of a filesystem-based path.

I kludged the Demux class' addParsers to strip the "hdfs://localhost:9000" portion of the string and now my class is found. I will attempt to provide a patch today that determines the value of Hadoop's fs.default.name and strips that from the value returned in Demux.java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CHUKWA-488) Hadoop cannot find custom Demux class

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CHUKWA-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870019#action_12870019 ] 

Hudson commented on CHUKWA-488:
-------------------------------

Integrated in Chukwa-trunk #373 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/373/])
    CHUKWA-488. Filter user customized jar file path URL.  (Kirk True via Eric Yang)


> Hadoop cannot find custom Demux class
> -------------------------------------
>
>                 Key: CHUKWA-488
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-488
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: MR Data Processors
>    Affects Versions: 0.4.0
>         Environment: Linux x86-64
> Java 1.6.0_20
>            Reporter: Kirk True
>             Fix For: 0.5.0
>
>         Attachments: Demux.diff
>
>
> I'm getting ClassNotFoundException errors when running inside Hadoop's map phase, unable to find my class org.apache.hadoop.chukwa.extraction.demux.processor.mapper.XmlBasedDemux which I've packaged in a JAR named data-collection-demux-0.1.jar.
> The problem seems to be in the values of these two properties in the Hadoop job configuration:
> {code}
> <property>
>     <name>mapred.job.classpath.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> <property>
>     <name>mapred.cache.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> {code}
> The problem seems to stem from the fact that the call to DistributedCache.addFileToClassPath is passing in a Path that is in URI form, i.e. hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar whereas the DistributedCache API expects it to be a filesystem-based path (i.e. /chukwa/demux/data-collection-demux-0.1.jar). I'm not sure why, but the FileStatus object returned by FileSystem.listStatus is returning a URL-based path instead of a filesystem-based path.
> I kludged the Demux class' addParsers to strip the "hdfs://localhost:9000" portion of the string and now my class is found. I will attempt to provide a patch today that determines the value of Hadoop's fs.default.name and strips that from the value returned in Demux.java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CHUKWA-488) Hadoop cannot find custom Demux class

Posted by "Kirk True (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CHUKWA-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirk True updated CHUKWA-488:
-----------------------------

    Attachment: Demux.diff

Not sure if this is the correct way to fix this, but it works on my development environment with default settings in Chukwa and Hadoop.

> Hadoop cannot find custom Demux class
> -------------------------------------
>
>                 Key: CHUKWA-488
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-488
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: MR Data Processors
>    Affects Versions: 0.4.0
>         Environment: Linux x86-64
> Java 1.6.0_20
>            Reporter: Kirk True
>         Attachments: Demux.diff
>
>
> I'm getting ClassNotFoundException errors when running inside Hadoop's map phase, unable to find my class org.apache.hadoop.chukwa.extraction.demux.processor.mapper.XmlBasedDemux which I've packaged in a JAR named data-collection-demux-0.1.jar.
> The problem seems to be in the values of these two properties in the Hadoop job configuration:
> {code}
> <property>
>     <name>mapred.job.classpath.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> <property>
>     <name>mapred.cache.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> {code}
> The problem seems to stem from the fact that the call to DistributedCache.addFileToClassPath is passing in a Path that is in URI form, i.e. hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar whereas the DistributedCache API expects it to be a filesystem-based path (i.e. /chukwa/demux/data-collection-demux-0.1.jar). I'm not sure why, but the FileStatus object returned by FileSystem.listStatus is returning a URL-based path instead of a filesystem-based path.
> I kludged the Demux class' addParsers to strip the "hdfs://localhost:9000" portion of the string and now my class is found. I will attempt to provide a patch today that determines the value of Hadoop's fs.default.name and strips that from the value returned in Demux.java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (CHUKWA-488) Hadoop cannot find custom Demux class

Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CHUKWA-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang resolved CHUKWA-488.
------------------------------

    Fix Version/s: 0.5.0
       Resolution: Fixed

I just committed this.  Thanks Kirk

> Hadoop cannot find custom Demux class
> -------------------------------------
>
>                 Key: CHUKWA-488
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-488
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: MR Data Processors
>    Affects Versions: 0.4.0
>         Environment: Linux x86-64
> Java 1.6.0_20
>            Reporter: Kirk True
>             Fix For: 0.5.0
>
>         Attachments: Demux.diff
>
>
> I'm getting ClassNotFoundException errors when running inside Hadoop's map phase, unable to find my class org.apache.hadoop.chukwa.extraction.demux.processor.mapper.XmlBasedDemux which I've packaged in a JAR named data-collection-demux-0.1.jar.
> The problem seems to be in the values of these two properties in the Hadoop job configuration:
> {code}
> <property>
>     <name>mapred.job.classpath.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> <property>
>     <name>mapred.cache.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> {code}
> The problem seems to stem from the fact that the call to DistributedCache.addFileToClassPath is passing in a Path that is in URI form, i.e. hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar whereas the DistributedCache API expects it to be a filesystem-based path (i.e. /chukwa/demux/data-collection-demux-0.1.jar). I'm not sure why, but the FileStatus object returned by FileSystem.listStatus is returning a URL-based path instead of a filesystem-based path.
> I kludged the Demux class' addParsers to strip the "hdfs://localhost:9000" portion of the string and now my class is found. I will attempt to provide a patch today that determines the value of Hadoop's fs.default.name and strips that from the value returned in Demux.java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CHUKWA-488) Hadoop cannot find custom Demux class

Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CHUKWA-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869728#action_12869728 ] 

Eric Yang commented on CHUKWA-488:
----------------------------------

+1 Looks good, and works on my test environment.

> Hadoop cannot find custom Demux class
> -------------------------------------
>
>                 Key: CHUKWA-488
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-488
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: MR Data Processors
>    Affects Versions: 0.4.0
>         Environment: Linux x86-64
> Java 1.6.0_20
>            Reporter: Kirk True
>         Attachments: Demux.diff
>
>
> I'm getting ClassNotFoundException errors when running inside Hadoop's map phase, unable to find my class org.apache.hadoop.chukwa.extraction.demux.processor.mapper.XmlBasedDemux which I've packaged in a JAR named data-collection-demux-0.1.jar.
> The problem seems to be in the values of these two properties in the Hadoop job configuration:
> {code}
> <property>
>     <name>mapred.job.classpath.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> <property>
>     <name>mapred.cache.files</name>
>     <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> {code}
> The problem seems to stem from the fact that the call to DistributedCache.addFileToClassPath is passing in a Path that is in URI form, i.e. hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar whereas the DistributedCache API expects it to be a filesystem-based path (i.e. /chukwa/demux/data-collection-demux-0.1.jar). I'm not sure why, but the FileStatus object returned by FileSystem.listStatus is returning a URL-based path instead of a filesystem-based path.
> I kludged the Demux class' addParsers to strip the "hdfs://localhost:9000" portion of the string and now my class is found. I will attempt to provide a patch today that determines the value of Hadoop's fs.default.name and strips that from the value returned in Demux.java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.