Crunch with Elastic MapReduce

[Shawn Smith <Sh...@bazaarvoice.com>]

Has anyone tried using Crunch with Amazon Elastic MapReduce?  I've run into a few issues, and I thought I'd share my experiences so far:

1. A typical Elastic MapReduce job uses S3 input and output files (w/Amazon's customized Native S3 File System) and HDFS intermediate files.  This doesn't work with Crunch calls to FileSystem.get(Configuration) that assume the default file system (HDFS).  Example stack trace:

Exception in thread "main" java.lang.IllegalArgumentException: This file system object (hdfs://10.114.37.65:9000) does not support access to the request path 's3://test-bucket/test/Input.avro' You possibly called FileSystem.get(conf) when you should have called FileSystem.get(uri, conf) to obtain a file system supporting your path.

        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:767)
        at org.apache.crunch.io.SourceTargetHelper.getPathSize(SourceTargetHelper.java:44)

It looks like switching to Path.getFileSystem(Configuration) throughout allows mixing S3 and HDFS files.

2. CrunchJob.handleMultiPaths() calls FileSystem.rename() which works only if the source and destination use the same file system.  This breaks the final upload of the output files from HDFS to S3.  I've created https://issues.apache.org/jira/browse/CRUNCH-47 and attached a patch with a crude workaround.  Note that when I ran into this the final jobs in my pipeline all failed, but without any error messages/stack traces in the logs pointing to the actual problem.  This might have been my fault because I hadn't called Pipeline.enableDebug() on that run.

3. EMR Hadoop 1.0.3 includes Avro 1.5.3 which apparently takes precedence over Crunch's Avro 1.7.0.  I didn't mess around with trying to get my classes in the class path first…  Instead I used the maven-shade-plugin in my job's build to shade Avro 1.7.0 from "org.apache.avro.*" to "shaded.org.apache.avro.*" so it wouldn't conflict with the EMR version of Avro.  Example exception (you can see the Avro source code line numbers correspond to version 1.5.3):

2012-08-13 06:50:57,547 WARN org.apache.hadoop.mapred.Child (main): Error running child
java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.avro.mapred.Pair.<init>()
at org.apache.avro.specific.SpecificDatumReader.newInstance(SpecificDatumReader.java:101)
at org.apache.avro.specific.SpecificDatumReader.newRecord(SpecificDatumReader.java:56)

4. EMR Hadoop 1.0.3 includes two different versions of SLF4J in the class path: 1.4.3 and 1.6.4.  As a result, jobs that use SLF4J will fail non-deterministically when a particular run uses slf4j-api-1.4.3.jar with slf4j-log4j12-1.6.4.jar, as described in the SLF4J FAQ (http://www.slf4j.org/faq.html#IllegalAccessError).  It looks like you can workaround the problem by using shaded SLF4J jars and not relying on the ones provided by the Hadoop distribution.  The stack trace looks something like this:

Exception in thread "main" java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory
at org.slf4j.LoggerFactory.<clinit>(LoggerFactory.java:60)
at com.bazaarvoice.etl.mapper.ClientMapper.<clinit>(ClientMapper.java:30)
at com.bazaarvoice.etl.hadoop.BulkTransformCommand.run(BulkTransformCommand.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.bazaarvoice.etl.hadoop.BulkTransformCommand.main(BulkTransformCommand.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)

Thanks,
Shawn Smith

Re: Crunch with Elastic MapReduce

[Matthias Friedrich <ma...@mafr.de>]

Hi Shawn,

thanks a lot for your mail and your patch! Answers see below.

On Tuesday, 2012-08-14, Shawn Smith wrote:
[...]
> 3. EMR Hadoop 1.0.3 includes Avro 1.5.3 which apparently takes precedence
> over Crunch's Avro 1.7.0.  I didn't mess around with trying to get my classes
> in the class path first…  Instead I used the maven-shade-plugin in my job's
> build to shade Avro 1.7.0 from "org.apache.avro.*" to
> "shaded.org.apache.avro.*" so it wouldn't conflict with the EMR version of
> Avro.  Example exception (you can see the Avro source code line numbers
> correspond to version 1.5.3):
[...]

Hadoop provides no classloader isolation, I've been bitten by this several
times, too. There's a crude workaround you can try:

    export HADOOP_USER_CLASSPATH_FIRST=true

You have to set it before running the hadoop script. I don't see other
options at this point until Hadoop is fixed.

[...]
> 4. EMR Hadoop 1.0.3 includes two different versions of SLF4J in the class path: 1.4.3 and 1.6.4.  As a result, jobs that use SLF4J will fail non-deterministically when a particular run uses slf4j-api-1.4.3.jar with slf4j-log4j12-1.6.4.jar, as described in the SLF4J FAQ (http://www.slf4j.org/faq.html#IllegalAccessError).  It looks like you can workaround the problem by using shaded SLF4J jars and not relying on the ones provided by the Hadoop distribution.  The stack trace looks something like this:

That's a nasty bug in EMR's Hadoop. We've recently downgraded our dependency
to slf4j 1.4.3 and I'm going to try fixing some more of these problems in
CRUNCH-16.

Please let us know if you experience any more problems!

Regards,
  Matthias