[jira] [Created] (CRUNCH-209) Jobs with large numbers of directory inputs will fail with odd inputsplit exceptions

["Josh Wills (JIRA)" <ji...@apache.org>]

Josh Wills created CRUNCH-209:
---------------------------------

             Summary: Jobs with large numbers of directory inputs will fail with odd inputsplit exceptions
                 Key: CRUNCH-209
                 URL: https://issues.apache.org/jira/browse/CRUNCH-209
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.6.0, 0.5.0
            Reporter: Josh Wills
            Assignee: Josh Wills
         Attachments: CRUNCH-209.patch

>From John Jensen on the user mailing list:

I have a curious problem when running a crunch job on (avro) files in a fairly large set of directories (just slightly less than 100).
After running some fraction of the mappers they start failing with the exception below. Things work fine with a smaller number of directories.

The magic 'zdHJpbmcifSx7Im5hbWUiOiJ2YWx1ZSIsInR5cGUiOiJzdHJpbmcifV19fSwiZGVmYXVsdCI' string shows up in the 'crunch.inputs.dir' entry in the job config, so I assume it has something to do with deserializing that value, but reading through the code I don't see any obvious way how. 

Furthermore, the crunch.inputs.dir config entry is just under 1.5M, so it would not surprise me if I'm running up against a hadoop limit somewhere.

Stack trace:

java.io.IOException: Split class zdHJp
bmcifSx7Im5hbWUiOiJ2YWx1ZSIsInR5cGUiOiJzdHJpbmcifV19fSwiZGVmYXVsdCI not found
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:342)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:614)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class zdHJp
bmcifSx7Im5hbWUiOiJ2YWx1ZSIsInR5cGUiOiJzdHJpbmcifV19fSwiZGVmYXVsdCI not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:340)
	... 7 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira