You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Vlad Rozov (JIRA)" <ji...@apache.org> on 2018/03/24 21:15:00 UTC
[jira] [Updated] (APEXMALHAR-2557) TimeBasedDedupOperator fails
with NullPointer
[ https://issues.apache.org/jira/browse/APEXMALHAR-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vlad Rozov updated APEXMALHAR-2557:
-----------------------------------
Description:
While using the TimeBasedDedupOperator for deduping, I see that operator
keeps failing with below NullPointer exception.
I also see that operator is always high on CPU usages. Almost reaching 100%.
No matter the setting any values for vcores or container memory for operator
{noformat}
2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler
(FSRecoveryHandler.java:rotateLog(103)) - Creating
hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log
2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent
(StreamingContainerParent.java:log(170)) - child msg: Stopped running due to
an exception. java.lang.NullPointerException
at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387)
at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66)
at
org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179)
at
org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139)
at
org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
context:
PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]]
2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager
(StreamingContainerManager.java:processOperatorFailure(1439)) - Operator
failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1
{noformat}
was:
While using the TimeBasedDedupOperator for deduping, I see that operator
keeps failing with below NullPointer exception.
I also see that operator is always high on CPU usages. Almost reaching 100%.
No matter the setting any values for vcores or container memory for operator
2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler
(FSRecoveryHandler.java:rotateLog(103)) - Creating
hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log
2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent
(StreamingContainerParent.java:log(170)) - child msg: Stopped running due to
an exception. java.lang.NullPointerException
at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387)
at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66)
at
org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179)
at
org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139)
at
org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
context:
PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]]
2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager
(StreamingContainerManager.java:processOperatorFailure(1439)) - Operator
failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1
> TimeBasedDedupOperator fails with NullPointer
> -----------------------------------------------
>
> Key: APEXMALHAR-2557
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2557
> Project: Apache Apex Malhar
> Issue Type: Bug
> Affects Versions: 3.8.0
> Reporter: Vivek Bhide
> Priority: Major
>
> While using the TimeBasedDedupOperator for deduping, I see that operator
> keeps failing with below NullPointer exception.
> I also see that operator is always high on CPU usages. Almost reaching 100%.
> No matter the setting any values for vcores or container memory for operator
> {noformat}
> 2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler
> (FSRecoveryHandler.java:rotateLog(103)) - Creating
> hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log
> 2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent
> (StreamingContainerParent.java:log(170)) - child msg: Stopped running due to
> an exception. java.lang.NullPointerException
> at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387)
> at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66)
> at
> org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179)
> at
> org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139)
> at
> org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> context:
> PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]]
> 2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager
> (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator
> failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)