You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2016/03/22 01:48:25 UTC
[jira] [Commented] (PIG-4845) Parallel instantiation of classes in
Tez cause tasks to fail
[ https://issues.apache.org/jira/browse/PIG-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205477#comment-15205477 ]
Daniel Dai commented on PIG-4845:
---------------------------------
+1
> Parallel instantiation of classes in Tez cause tasks to fail
> ------------------------------------------------------------
>
> Key: PIG-4845
> URL: https://issues.apache.org/jira/browse/PIG-4845
> Project: Pig
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.16.0
>
> Attachments: PIG-4845-1.patch
>
>
> OptionBuilder is not thread-safe
> https://commons.apache.org/proper/commons-cli/javadocs/api-release/org/apache/commons/cli/OptionBuilder.html
> {code}
> This class is NOT thread safe. See CLI-209
> {code}
> HBaseStorage, elephantbird SequenceFileConfig, etc use it in their constructor. This leads to NoSuchMethodException, UnrecognizedOptionException etc when processor, inputs and outputs are initialized in parallel in Tez making the task fail. Retry attempts mostly go through and job might succeed. Need to synchronize the initialization. synchronize would make it similar to mapreduce though at a slight performance cost. But that should be lot better than cost of relaunching containers after hitting failures.
> Following two kinds of exception where seen when HBaseStorage and SequenceFileStorage of elephantbird were used together.
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -loadKey
> {code}
> Caused by: java.lang.RuntimeException: Failed to create WritableConverter instance
> at com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:225)
> at com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:101)
> at com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:115)
> ... 11 more
> Caused by: java.lang.NoSuchMethodException: com.twitter.elephantbird.pig.util.TextConverter.<init>(java.lang.String)
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.getConstructor(Class.java:1825)
> at com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:213)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)