You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2010/02/22 20:20:28 UTC
[jira] Created: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
CompressionCodecFactory throws IllegalArgumentException
-------------------------------------------------------
Key: HADOOP-6588
URL: https://issues.apache.org/jira/browse/HADOOP-6588
Project: Hadoop Common
Issue Type: Bug
Components: io
Reporter: Tsz Wo (Nicholas), SZE
WordCount does not run. :(
{noformat}
java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
...
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842072#action_12842072 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-6588:
------------------------------------------------
> That is because you specified LzoCodec class in your conf file. ...
It probably is the case.
> ... I think it is better to leave the behavior as-is instead of issuing log messages.
I am fine for keeping the current behavior, i.e. throwing an exception. However, it is better to change the exception since it is not an IllegalArgumentException.
Also, CompressionCodecFactory constructor calls getCodecClasses(..) which ultimately calls Class.forName(..). As a consequence, Class.forName(..) will be invoked for every TextInputFormat.isSplitable(..). It is unnecessary and expensive.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842104#action_12842104 ]
Hong Tang commented on HADOOP-6588:
-----------------------------------
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837023#action_12837023 ]
Hong Tang commented on HADOOP-6588:
-----------------------------------
That is because you specified LzoCodec class in your conf file. I think it is better to leave the behavior as-is instead of issuing log messages.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842121#action_12842121 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-6588:
------------------------------------------------
How about caching the classes in Configuration.getClassByName(..)? It seems a right place since Configuration also manages a classLoader.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-6588:
-------------------------------------------
Attachment: c6588_20100222.patch
c6588_20100222.patch: replaced IllegalArgumentException by a log message.
BTW, I also suggest to have a static variable storing the CompressionCodec list since the operations in CompressionCodecFactory.getCodecClasses(..) are quite expensive.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842092#action_12842092 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-6588:
------------------------------------------------
> I assume you mean o.a.h.mapreduce.TextInputFormat. Not sure how to fix it in CompressionCodecFactory though. ...
It should be org.apache.hadoop.mapreduce.lib.input.TextInputFormat.
I think the CompressionCodecFactory should cache the codec classes and only calls Class.forName(..) at the first time.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842079#action_12842079 ]
Hong Tang commented on HADOOP-6588:
-----------------------------------
bq. I am fine for keeping the current behavior, i.e. throwing an exception. However, it is better to change the exception since it is not an IllegalArgumentException.
Makes sense, probably should let it throw ClassNotFoundException.
bq. Also, CompressionCodecFactory constructor calls getCodecClasses(..) which ultimately calls Class.forName(..). As a consequence, Class.forName(..) will be invoked for every TextInputFormat.isSplitable(..). It is unnecessary and expensive.
I assume you mean o.a.h.mapreduce.TextInputFormat. Not sure how to fix it in CompressionCodecFactory though. Probably be better to cache an instance of the factory in TextInputFormat, but the semantics is slightly different from the current code - if different conf objects are passed down to multiple invocations of this method.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6588) CompressionCodecFactory throws
IllegalArgumentException
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842102#action_12842102 ]
Hong Tang commented on HADOOP-6588:
-----------------------------------
bq. I think the CompressionCodecFactory should cache the codec classes and only calls Class.forName(..) at the first time.
I generally do not like singleton caches. And I think caching the factory at TextInputFormat level is probably better, as it also saves the cost of creating a factory object for each isSplitable call. In fact, the o.a.h.mapred.TextInputFormat does exactly that.
> CompressionCodecFactory throws IllegalArgumentException
> -------------------------------------------------------
>
> Key: HADOOP-6588
> URL: https://issues.apache.org/jira/browse/HADOOP-6588
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments: c6588_20100222.patch
>
>
> WordCount does not run. :(
> {noformat}
> java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:247)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:886)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:780)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:444)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:459)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.