You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2023/03/03 08:02:12 UTC
[GitHub] [druid] clintropolis opened a new issue, #13879: Sampler API ClassCastException when using KafkaInputFormat
clintropolis opened a new issue, #13879:
URL: https://github.com/apache/druid/issues/13879
Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
### Affected Version
0.23.0+
### Description
The `KafkaInputFormat` added in #11630 allows extracting additional data from Kafka messages to ingest into Druid. However, it has a problem with the Druid sampler API, exploding with:
```
2023-03-03T07:36:02,271 ERROR [qtp1379563521-109] org.apache.druid.indexing.overlord.sampler.SamplerExceptionMapper - Failed to sample data: org.apache.druid.data.input.kafka.KafkaRecordEntity cannot be cast to org.apache.druid.indexing.seekablestream.SettableByteEntity
org.apache.druid.indexing.overlord.sampler.SamplerException: Failed to sample data: org.apache.druid.data.input.kafka.KafkaRecordEntity cannot be cast to org.apache.druid.indexing.seekablestream.SettableByteEntity
at org.apache.druid.indexing.overlord.sampler.InputSourceSampler.sample(InputSourceSampler.java:281) ~[classes/:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamSamplerSpec.sample(SeekableStreamSamplerSpec.java:116) ~[classes/:?]
at org.apache.druid.indexing.overlord.sampler.SamplerResource.post(SamplerResource.java:43) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
...
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_332]
Caused by: java.lang.ClassCastException: org.apache.druid.data.input.kafka.KafkaRecordEntity cannot be cast to org.apache.druid.indexing.seekablestream.SettableByteEntity
at org.apache.druid.data.input.kafkainput.KafkaInputFormat.createReader(KafkaInputFormat.java:84) ~[?:?]
at org.apache.druid.data.input.impl.InputEntityIteratingReader.lambda$sample$1(InputEntityIteratingReader.java:97) ~[classes/:?]
at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:84) ~[classes/:?]
at org.apache.druid.java.util.common.parsers.CloseableIterator$2.<init>(CloseableIterator.java:69) ~[classes/:?]
at org.apache.druid.java.util.common.parsers.CloseableIterator.flatMap(CloseableIterator.java:67) ~[classes/:?]
at org.apache.druid.data.input.impl.InputEntityIteratingReader.createIterator(InputEntityIteratingReader.java:108) ~[classes/:?]
at org.apache.druid.data.input.impl.InputEntityIteratingReader.sample(InputEntityIteratingReader.java:94) ~[classes/:?]
at org.apache.druid.data.input.impl.TimedShutoffInputSourceReader.sample(TimedShutoffInputSourceReader.java:62) ~[classes/:?]
at org.apache.druid.segment.transform.TransformingInputSourceReader.sample(TransformingInputSourceReader.java:50) ~[classes/:?]
at org.apache.druid.indexing.overlord.sampler.InputSourceSampler.sample(InputSourceSampler.java:129) ~[classes/:?]
```
which is also what powers the web-console data loader. This is because it is making some assumptions about a `SettableByteEntityReader` being involved which it can use to do its thing.
This can be resolved by modifying `KafkaInputFormat.createReader` to something like
```
if (source instanceof SettableByteEntity) {
settableByteEntitySource = (SettableByteEntity<KafkaRecordEntity>) source;
} else {
settableByteEntitySource = new SettableByteEntity<>();
settableByteEntitySource.setEntity((KafkaRecordEntity) source);
}
```
to wrap the entity in a `SettableByteEntity` needed by the `KafkaInputReader` which allows the sampler (and data loader ui) to function correctly:
<img width="968" alt="Screenshot 2023-03-03 at 12 01 33 AM" src="https://user-images.githubusercontent.com/1577461/222664867-c19476ae-c60e-49ef-b215-d9a92199fd99.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm closed issue #13879: Sampler API ClassCastException when using KafkaInputFormat
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm closed issue #13879: Sampler API ClassCastException when using KafkaInputFormat
URL: https://github.com/apache/druid/issues/13879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org