You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Mikhailau, Alex" <Al...@mlb.com> on 2017/10/04 14:43:17 UTC
Re: Re-sharded kinesis stream starts generating warnings after
kinesis shard numbers were doubled
Just found the same exact issues in one of our long running clusters:
Cannot get the shard for this ProcessTask, so duplicate KPL user records in the event of resharding will not be dropped during deaggregation of Amazon Kinesis records.
Cannot find the shard given the shardId shardId-000000004454
With 2.2.0
-Alex
From: "Mikhailau, Alex" <Al...@mlb.com>
Date: Wednesday, September 13, 2017 at 4:16 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re-sharded kinesis stream starts generating warnings after kinesis shard numbers were doubled
Has anyone seen the following warnings in the log after a kinesis stream has been re-sharded?
com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask
WARN Cannot get the shard for this ProcessTask, so duplicate KPL user records in the event of resharding will not be dropped during deaggregation of Amazon Kinesis records.
com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy
WARN Cannot find the shard given the shardId shardId-000000000599
Re: Re-sharded kinesis stream starts generating warnings after
kinesis shard numbers were doubled
Posted by "Mikhailau, Alex" <Al...@mlb.com>.
Filed SPARK-22200
From: "Mikhailau, Alex" <Al...@mlb.com>
Date: Wednesday, October 4, 2017 at 10:43 AM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re: Re-sharded kinesis stream starts generating warnings after kinesis shard numbers were doubled
Just found the same exact issues in one of our long running clusters:
Cannot get the shard for this ProcessTask, so duplicate KPL user records in the event of resharding will not be dropped during deaggregation of Amazon Kinesis records.
Cannot find the shard given the shardId shardId-000000004454
With 2.2.0
-Alex
From: "Mikhailau, Alex" <Al...@mlb.com>
Date: Wednesday, September 13, 2017 at 4:16 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re-sharded kinesis stream starts generating warnings after kinesis shard numbers were doubled
Has anyone seen the following warnings in the log after a kinesis stream has been re-sharded?
com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask
WARN Cannot get the shard for this ProcessTask, so duplicate KPL user records in the event of resharding will not be dropped during deaggregation of Amazon Kinesis records.
com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy
WARN Cannot find the shard given the shardId shardId-000000000599