You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "jia.fu (JIRA)" <ji...@apache.org> on 2015/11/28 07:23:11 UTC
[jira] [Commented] (STORM-643) KafkaUtils repeatedly fetches
messages whose offset is out of range
[ https://issues.apache.org/jira/browse/STORM-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15030404#comment-15030404 ]
jia.fu commented on STORM-643:
------------------------------
I'm using this ver storm-kafka 0.95.and got a large number of logs like below .
and set up spoutConf.forceFromStart = true;
the root cause this jira?
2015-11-28T12:39:28.843+0800 s.k.PartitionManager [WARN] Using new offset: 26147475
2015-11-28T12:39:28.850+0800 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [9852196]; retrying with default start offset time from configuration. configured start offset time: [-2]
2015-11-28T12:39:28.850+0800 s.k.PartitionManager [WARN] Using new offset: 26147475
2015-11-28T12:39:28.851+0800 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [9852196]; retrying with default start offset time from configuration. configured start offset time: [-2]
2015-11-28T12:39:28.852+0800 s.k.PartitionManager [WARN] Using new offset: 26147475
2015-11-28T12:39:28.853+0800 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [9852196]; retrying with default start offset time from configuration. configured start offset time: [-2]
2015-11-28T12:39:28.853+0800 s.k.PartitionManager [WARN] Using new offset: 26147475
2015-11-28T12:39:28.855+0800 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [9852196]; retrying with default start offset time from configuration. configured start offset time: [-2]
2015-11-28T12:39:28.855+0800 s.k.PartitionManager [WARN] Using new offset: 26147475
> KafkaUtils repeatedly fetches messages whose offset is out of range
> -------------------------------------------------------------------
>
> Key: STORM-643
> URL: https://issues.apache.org/jira/browse/STORM-643
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
> Affects Versions: 0.9.2-incubating, 0.9.3, 0.10.0, 0.9.4, 0.9.5
> Reporter: Xin Wang
> Assignee: Xin Wang
> Fix For: 0.9.6
>
>
> KafkaUtils repeat fetch messages which offset is out of range.
> This happened when failed list(SortedSet<Long> failed) is not empty and some offset in it is OutOfRange.
> [worker-log]
> {code}
> 2015-02-01 10:24:27.231+0800 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [20919071816]; retrying with default start offset time from configuration. configured start offset time: [-2]
> 2015-02-01 10:24:27.232+0800 s.k.PartitionManager [WARN] Using new offset: 20996130717
> 2015-02-01 10:24:27.333+0800 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [20919071816]; retrying with default start offset time from configuration. configured start offset time: [-2]
> 2015-02-01 10:24:27.334+0800 s.k.PartitionManager [WARN] Using new offset: 20996130717
> ...
> {code}
> [FIX]
> {code}
> storm.kafka.PartitionManager.fill():
> ...
> try {
> msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);
> } catch (UpdateOffsetException e) {
> _emittedToOffset = KafkaUtils.getOffset(_consumer, _spoutConfig.topic, _partition.partition, _spoutConfig);
> LOG.warn("Using new offset: {}", _emittedToOffset);
> // fetch failed, so don't update the metrics
> //fix bug: remove this offset from failed list when it is OutOfRange
> if (had_failed) {
> failed.remove(offset);
> }
> return;
> }
> ...
> {code}
> also: Log "retrying with default start offset time from configuration. configured start offset time: [-2]" is incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)