You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by "Victor Iacoban (JIRA)" <ji...@apache.org> on 2012/10/29 23:19:11 UTC

[jira] [Created] (CRUNCH-104) upgrade slf4j

Victor Iacoban created CRUNCH-104:
-------------------------------------

             Summary: upgrade slf4j
                 Key: CRUNCH-104
                 URL: https://issues.apache.org/jira/browse/CRUNCH-104
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.3.0
         Environment: Linux
            Reporter: Victor Iacoban
            Assignee: Josh Wills


Crunch is using an ancient version of slf4j, making it impossible to integrate in environment using newer slf4j
Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
        at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:136)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:222)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:201)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:276)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:473)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1668)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1661)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1529)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:212)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
        at org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:270)
        at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:95)
        at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:86)
        at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:78)
        at weka.ExportARFF.main(ExportARFF.java:44)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CRUNCH-104) upgrade slf4j

Posted by "Matthias Friedrich (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/CRUNCH-104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Friedrich resolved CRUNCH-104.
---------------------------------------

       Resolution: Invalid
    Fix Version/s: 0.4.0

Hi Victor, we're aware of the problem and really, really frustrated because of it but I'm afraid we can't help you here. Hadoop 1.0.3 has slf4j-1.4.3 on its classpath, and that's what is used by MR jobs, no matter which version of slf4j Crunch or your job ships. One workaround for Hadoop 1.0.3 is to override slf4j in your POM with a later version and set the HADOOP_USER_CLASSPATH_FIRST environment variable to true when executing your job.

If you want to help you can open an issue with Hadoop, the more people complain about this, the better.
                
> upgrade slf4j
> -------------
>
>                 Key: CRUNCH-104
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-104
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.3.0
>         Environment: Linux
>            Reporter: Victor Iacoban
>            Assignee: Josh Wills
>             Fix For: 0.4.0
>
>
> Crunch is using an ancient version of slf4j, making it impossible to integrate in environment using newer slf4j
> Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
>         at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
>         at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:136)
>         at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:222)
>         at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:201)
>         at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:276)
>         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
>         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:473)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1668)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1661)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1529)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:212)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>         at org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:270)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:95)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:86)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:78)
>         at weka.ExportARFF.main(ExportARFF.java:44)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CRUNCH-104) upgrade slf4j

Posted by "Josh Wills (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/CRUNCH-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486458#comment-13486458 ] 

Josh Wills commented on CRUNCH-104:
-----------------------------------

Hey Victor, [~mafr] handles our dependency management-- we try to be sure our dependencies play nicely with each other, but that can often lead to having to support the lowest-common denominator. Have you tried using maven's <excludes> feature to not include Crunch's slf4j dependency?
                
> upgrade slf4j
> -------------
>
>                 Key: CRUNCH-104
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-104
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.3.0
>         Environment: Linux
>            Reporter: Victor Iacoban
>            Assignee: Josh Wills
>
> Crunch is using an ancient version of slf4j, making it impossible to integrate in environment using newer slf4j
> Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
>         at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
>         at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:136)
>         at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:222)
>         at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:201)
>         at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:276)
>         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
>         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:473)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1668)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1661)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1529)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:212)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>         at org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:270)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:95)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:86)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:78)
>         at weka.ExportARFF.main(ExportARFF.java:44)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CRUNCH-104) upgrade slf4j

Posted by "Victor Iacoban (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/CRUNCH-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486829#comment-13486829 ] 

Victor Iacoban commented on CRUNCH-104:
---------------------------------------

Josh, Matthias,

I was able to make it run by overriding/downgrading slf4j version to 1.4.3

Thanks and keep up the good work
                
> upgrade slf4j
> -------------
>
>                 Key: CRUNCH-104
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-104
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.3.0
>         Environment: Linux
>            Reporter: Victor Iacoban
>            Assignee: Josh Wills
>             Fix For: 0.4.0
>
>
> Crunch is using an ancient version of slf4j, making it impossible to integrate in environment using newer slf4j
> Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
>         at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
>         at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:136)
>         at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:222)
>         at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:201)
>         at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:276)
>         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
>         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:473)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1668)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1661)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1529)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:212)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>         at org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:270)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:95)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:86)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:78)
>         at weka.ExportARFF.main(ExportARFF.java:44)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CRUNCH-104) upgrade slf4j

Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/CRUNCH-104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabriel Reid updated CRUNCH-104:
--------------------------------

    Fix Version/s:     (was: 0.4.0)
    
> upgrade slf4j
> -------------
>
>                 Key: CRUNCH-104
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-104
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.3.0
>         Environment: Linux
>            Reporter: Victor Iacoban
>            Assignee: Josh Wills
>
> Crunch is using an ancient version of slf4j, making it impossible to integrate in environment using newer slf4j
> Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
>         at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
>         at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:136)
>         at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:222)
>         at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:201)
>         at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:276)
>         at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
>         at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:473)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1668)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1661)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1529)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:212)
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>         at org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:270)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:95)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:86)
>         at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:78)
>         at weka.ExportARFF.main(ExportARFF.java:44)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira