You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "DiogoP98 (via GitHub)" <gi...@apache.org> on 2026/05/06 15:28:35 UTC
[I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
DiogoP98 opened a new issue, #8588:
URL: https://github.com/apache/storm/issues/8588
### Description
After upgrading from Apache Storm 2.8.0 to 2.8.7, the "Kafka Spouts Lag" panel in the Storm UI stops working for many topologies. The UI displays no lag information for affected spouts, and `nimbus.log` / `ui.log` are flooded with `ClassCastException` warnings (with a null exception message) for every Kafka spout in those topologies.
### To Reproduce
1. Run a topology containing one or more `KafkaSpout` instances.
2. Configure at least one of the spout's subscribed topic-partitions so that no offset has been committed yet for the consumer group (e.g., a brand-new consumer group, a passive replication topic, or an idle partition).
3. Open the topology page in Storm UI and look at the "Kafka Spouts Lag" panel.
4. Observe that the panel shows no data for that spout.
5. Inspect the UI/Nimbus logs.
### Expected behavior
The lag panel reports per-partition lag for each Kafka spout. Partitions with no committed offset are reported gracefully (e.g., as -1) without breaking the rest of the spout's lag query. No `ClassCastException` warnings appear in the logs.
### Actual behavior
For every affected spout, the following warning is logged on each lag refresh:
```
TopologySpoutLag [WARN] Exception thrown while getting lag for spout id: <spoutId>
TopologySpoutLag [WARN] Exception message:null
java.lang.ClassCastException
```
The spoutLagResult for those spouts is missing entirely from the `/api/v1/topology/<id>/lag` response, and the UI's "Kafka Spouts Lag" panel stays empty.
### Identified Root cause
There are two interacting bugs introduced by the Kafka client upgrade from 3.9.0 to 4.x (commit 2958d3ff8):
1. **KafkaOffsetLagUtil.getOffsetLags** (`storm-kafka-monitor`): the [Kafka 4 API replaced](https://kafka.apache.org/40/getting-started/upgrade/#:~:text=timeout%2E-,The,instead) **consumer.committed(TopicPartition)** (which returned OffsetAndMetadata or null) with **consumer.committed(Set<TopicPartition>)** (which returns a Map). The migration kept the old null-check shape but on the wrong object:
```
Map<TopicPartition, OffsetAndMetadata> offsetAndMetadata = consumer.committed(Collections.singleton(topicPartition));
long committedOffset = offsetAndMetadata != null ? offsetAndMetadata.get(topicPartition).offset() : -1;
```
2. The map itself is never null, but its value for a partition without a committed offset is null. .offset() on null throws `NullPointerException`, which is caught by main().
3. **TopologySpoutLag.getLagResultForKafka** (`storm-core`): captures the monitor's `stdout` and parses it with `JSONValue.parseWithException`, expecting either a JSON object or a ParseException. However, json-smart parses unquoted plain text leniently as a java.lang.String rather than throwing ParseException. The unchecked cast (Map<String, Object>) parsedResult then throws ClassCastException (with null detail message on some JVMs), which is caught by the outer handler in lag() and surfaces as the warning above. The original error text from the monitor is lost.
In previous Storm version with Kafka client 3.9.0, the `committed(TopicPartition)` API correctly returned null, the existing null-check worked, and the chain never broke, so this only surfaces after the 4.x upgrade.
### Proposed fix
1. In KafkaOffsetLagUtil, null-check the value retrieved from the map, not the map itself:
```
Map<TopicPartition, OffsetAndMetadata> committed = consumer.committed(Collections.singleton(topicPartition));
OffsetAndMetadata partitionOffset = committed.get(topicPartition);
long committedOffset = partitionOffset != null ? partitionOffset.offset() : -1;
```
2. In TopologySpoutLag, defensively handle the case where `JSONValue.parseWithException` succeeds but yields a non-Map (e.g., a String from leniently parsed error text). Treat that as a monitor failure and surface the raw stdout as the spout's errorInfo, instead of letting the cast throw.
This way uncommitted partitions report -1 instead of crashing the whole spout's query, and any future monitor error is propagated to the UI as a useful error message rather than silently lost.
### Environment
- Apache Storm 2.8.7
- Java 17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on issue #8588:
URL: https://github.com/apache/storm/issues/8588#issuecomment-4389658283
Thanks for the Report. Feel free to propose a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu closed issue #8588: Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag
URL: https://github.com/apache/storm/issues/8588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on issue #8588:
URL: https://github.com/apache/storm/issues/8588#issuecomment-4389951292
Yes, Kafka lacks coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on issue #8588:
URL: https://github.com/apache/storm/issues/8588#issuecomment-4410279533
backport for 2.x : https://github.com/apache/storm/pull/8591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
Posted by "DiogoP98 (via GitHub)" <gi...@apache.org>.
DiogoP98 commented on issue #8588:
URL: https://github.com/apache/storm/issues/8588#issuecomment-4392209931
PR created: https://github.com/apache/storm/pull/8589. I've also added an integration test for this case. Let me know if you'd like more depth in these tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kafka spout lag broken after Kafka 4 client upgrade — `ClassCastException` in TopologySpoutLag (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on issue #8588:
URL: https://github.com/apache/storm/issues/8588#issuecomment-4389925791
Thanks for that. It seems we missed a simple test for that one. I can also
fix it if you want.
Cheers
On Wed, 6 May 2026 at 16:34, Richard Zowalla ***@***.***>
wrote:
> *rzo1* left a comment (apache/storm#8588)
> <https://github.com/apache/storm/issues/8588#issuecomment-4389658283>
>
> Thanks for the Report. Feel free to propose a PR.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/storm/issues/8588#issuecomment-4389658283>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAG5GITYJ23D3ANYAFLBPBL4ZNLRXAVCNFSM6AAAAACYTINQEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGOBZGY2TQMRYGM>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org