You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by patcharee <Pa...@uni.no> on 2015/05/19 11:19:46 UTC
conflict from apache commons codec
Hi,
I am using Pig version 0.14 with tez version 0.5.2. I have apache
commons-codec-1.4.jar on the machine. However, I got the common codec
exception when I executed a Pig job with tez.
2015-05-19 11:01:04,784 INFO [AsyncDispatcher event handler]
history.HistoryEventHandler:
[HISTORY][DAG:dag_1431972385685_0021_1][Event:TASK_ATTEMPT_FINISHED]:
vertexName=scope-384,
taskAttemptId=attempt_1431972385685_0021_1_00_000004_0,
startTime=1432025450752, finishTime=1432026064784, timeTaken=614032,
status=FAILED, diagnostics=Error: Fatal Error cause TezChild
exit.:java.lang.NoSuchMethodError:
org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B
at
org.apache.hadoop.yarn.util.AuxiliaryServiceHelper.getServiceDataFromEnv(AuxiliaryServiceHelper.java:37)
at
org.apache.tez.runtime.api.impl.TezTaskContextImpl.getServiceProviderMetaData(TezTaskContextImpl.java:175)
at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.generateEventsOnClose(OrderedPartitionedKVOutput.java:187)
at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:148)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:348)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:178)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
What could be wrong?
BR,
Patcharee
Re: conflict from apache commons codec
Posted by Siddharth Seth <ss...@apache.org>.
commons-codec contains the method since 1.4 - so those two versions of the
jar in the classpath should not cause issues.
Are there any custom jars which are loaded ? I'd look at any fat jars (Pig
for example) - to see if they're bundling commons-codec within them.
You could try this programmatically figure out which jar is being used for
the specific class - will likely require writing a custom function for Pig.
For small vs large jobs - it is likely the order in which jars are picked
up on different containers. I'm guessing the larger job has more containers
- which increases the chances of some container picking up the wrong jar.
Is this a multi-node cluster ?, and are any jars being picked up from the
cluster itself (instead of everything via local resources). Looking at
specific nodes for an older version would be another thing to look at.
On Wed, May 20, 2015 at 3:08 AM, patcharee <Pa...@uni.no>
wrote:
> Thanks for your input. There are two weird points found:
>
> - In my cluster (also the resources being localized for the tasks) there
> are two versions; commons-codec-1.4.jar and commons-codec-1.7.jar. I think
> commons-codec since version 1.4 shows be fine, but I still got the
> exception (NoSuchMethodError
> org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B).
> Any ideas?
>
> - I tried to run the same pig script+same environment on small number of
> files and large number of files. The former (small number of files) did not
> throw the exception, but the latter did. What can be wrong for the latter?
>
> BR,
> Patcharee
>
>
>
> On 20. mai 2015 09:37, Siddharth Seth wrote:
>
> My best guess would be that an older version of commons-codec is also on
> the classpath for the running task. If you have access to the local-dirs
> configured under YARN - you could find the application dir in the
> local-dirs and see what exists in the classpath for a container.
>
> Alternately, set tez.generate.debug.artifacts to true. This should give
> you access to the dag plan in text form via the YARN UI - which will list
> out the resources being localized for tasks.
>
> On Tue, May 19, 2015 at 2:19 AM, patcharee <Pa...@uni.no>
> wrote:
>
>> Hi,
>>
>> I am using Pig version 0.14 with tez version 0.5.2. I have apache
>> commons-codec-1.4.jar on the machine. However, I got the common codec
>> exception when I executed a Pig job with tez.
>>
>> 2015-05-19 11:01:04,784 INFO [AsyncDispatcher event handler]
>> history.HistoryEventHandler:
>> [HISTORY][DAG:dag_1431972385685_0021_1][Event:TASK_ATTEMPT_FINISHED]:
>> vertexName=scope-384,
>> taskAttemptId=attempt_1431972385685_0021_1_00_000004_0,
>> startTime=1432025450752, finishTime=1432026064784, timeTaken=614032,
>> status=FAILED, diagnostics=Error: Fatal Error cause TezChild
>> exit.:java.lang.NoSuchMethodError:
>> org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B
>> at
>> org.apache.hadoop.yarn.util.AuxiliaryServiceHelper.getServiceDataFromEnv(AuxiliaryServiceHelper.java:37)
>> at
>> org.apache.tez.runtime.api.impl.TezTaskContextImpl.getServiceProviderMetaData(TezTaskContextImpl.java:175)
>> at
>> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.generateEventsOnClose(OrderedPartitionedKVOutput.java:187)
>> at
>> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:148)
>> at
>> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:348)
>> at
>> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:178)
>> at
>> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> at
>> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
>> at
>> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> What could be wrong?
>>
>> BR,
>> Patcharee
>>
>>
>>
>>
>
>
Re: conflict from apache commons codec
Posted by patcharee <Pa...@uni.no>.
Thanks for your input. There are two weird points found:
- In my cluster (also the resources being localized for the tasks) there
are two versions; commons-codec-1.4.jar and commons-codec-1.7.jar. I
think commons-codec since version 1.4 shows be fine, but I still got the
exception (NoSuchMethodError
org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B).
Any ideas?
- I tried to run the same pig script+same environment on small number of
files and large number of files. The former (small number of files) did
not throw the exception, but the latter did. What can be wrong for the
latter?
BR,
Patcharee
On 20. mai 2015 09:37, Siddharth Seth wrote:
> My best guess would be that an older version of commons-codec is also
> on the classpath for the running task. If you have access to the
> local-dirs configured under YARN - you could find the application dir
> in the local-dirs and see what exists in the classpath for a container.
>
> Alternately, set tez.generate.debug.artifacts to true. This should
> give you access to the dag plan in text form via the YARN UI - which
> will list out the resources being localized for tasks.
>
> On Tue, May 19, 2015 at 2:19 AM, patcharee <Patcharee.Thongtra@uni.no
> <ma...@uni.no>> wrote:
>
> Hi,
>
> I am using Pig version 0.14 with tez version 0.5.2. I have apache
> commons-codec-1.4.jar on the machine. However, I got the common
> codec exception when I executed a Pig job with tez.
>
> 2015-05-19 11:01:04,784 INFO [AsyncDispatcher event handler]
> history.HistoryEventHandler:
> [HISTORY][DAG:dag_1431972385685_0021_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=scope-384,
> taskAttemptId=attempt_1431972385685_0021_1_00_000004_0,
> startTime=1432025450752, finishTime=1432026064784,
> timeTaken=614032, status=FAILED, diagnostics=Error: Fatal Error
> cause TezChild exit.:java.lang.NoSuchMethodError:
> org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B
> at
> org.apache.hadoop.yarn.util.AuxiliaryServiceHelper.getServiceDataFromEnv(AuxiliaryServiceHelper.java:37)
> at
> org.apache.tez.runtime.api.impl.TezTaskContextImpl.getServiceProviderMetaData(TezTaskContextImpl.java:175)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.generateEventsOnClose(OrderedPartitionedKVOutput.java:187)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:148)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:178)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> What could be wrong?
>
> BR,
> Patcharee
>
>
>
>
Re: conflict from apache commons codec
Posted by Siddharth Seth <ss...@apache.org>.
My best guess would be that an older version of commons-codec is also on
the classpath for the running task. If you have access to the local-dirs
configured under YARN - you could find the application dir in the
local-dirs and see what exists in the classpath for a container.
Alternately, set tez.generate.debug.artifacts to true. This should give you
access to the dag plan in text form via the YARN UI - which will list out
the resources being localized for tasks.
On Tue, May 19, 2015 at 2:19 AM, patcharee <Pa...@uni.no>
wrote:
> Hi,
>
> I am using Pig version 0.14 with tez version 0.5.2. I have apache
> commons-codec-1.4.jar on the machine. However, I got the common codec
> exception when I executed a Pig job with tez.
>
> 2015-05-19 11:01:04,784 INFO [AsyncDispatcher event handler]
> history.HistoryEventHandler:
> [HISTORY][DAG:dag_1431972385685_0021_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=scope-384,
> taskAttemptId=attempt_1431972385685_0021_1_00_000004_0,
> startTime=1432025450752, finishTime=1432026064784, timeTaken=614032,
> status=FAILED, diagnostics=Error: Fatal Error cause TezChild
> exit.:java.lang.NoSuchMethodError:
> org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B
> at
> org.apache.hadoop.yarn.util.AuxiliaryServiceHelper.getServiceDataFromEnv(AuxiliaryServiceHelper.java:37)
> at
> org.apache.tez.runtime.api.impl.TezTaskContextImpl.getServiceProviderMetaData(TezTaskContextImpl.java:175)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.generateEventsOnClose(OrderedPartitionedKVOutput.java:187)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:148)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:348)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:178)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> What could be wrong?
>
> BR,
> Patcharee
>
>
>
>