You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "JeffBolle (via GitHub)" <gi...@apache.org> on 2023/09/06 18:14:47 UTC
[GitHub] [pinot] JeffBolle opened a new issue, #11530: Protobuf Stream Config validationg assumes Kafka streams
JeffBolle opened a new issue, #11530:
URL: https://github.com/apache/pinot/issues/11530
https://github.com/apache/pinot/blob/e949d958a8238a48ba147048fa72ada033ea889e/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java#L518
This function assumes that the streaming ingest will be `kafka`. Something like this might work better:
```
static void validateDecoder(StreamConfig streamConfig) {
if (streamConfig.getDecoderClass().equals("org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder")) {
String streamType = streamConfig.getType();
String descriptorFilePath = String.format("stream.%s.decoder.prop.descriptorFile", streamType);
String protoClassName = String.format("stream.%s.decoder.prop.protoClassName",streamType);
// check the existence of the needed decoder props
if (!streamConfig.getDecoderProperties().containsKey(descriptorFilePath)) {
throw new IllegalStateException("Missing property of descriptorFile for ProtoBufMessageDecoder");
}
if (!streamConfig.getDecoderProperties().containsKey(protoClassName)) {
throw new IllegalStateException("Missing property of protoClassName for ProtoBufMessageDecoder");
}
}
}
```
@navina
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] JeffBolle commented on issue #11530: Protobuf Stream Config validationg assumes Kafka streams
Posted by "JeffBolle (via GitHub)" <gi...@apache.org>.
JeffBolle commented on issue #11530:
URL: https://github.com/apache/pinot/issues/11530#issuecomment-1710350230
Maybe...I'm working another issue with missing metadata that is more important in my environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 commented on issue #11530: Protobuf Stream Config validationg assumes Kafka streams
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on issue #11530:
URL: https://github.com/apache/pinot/issues/11530#issuecomment-1709300424
Good catch, @JeffBolle do you want to open a PR to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 closed issue #11530: Protobuf Stream Config validationg assumes Kafka streams
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed issue #11530: Protobuf Stream Config validationg assumes Kafka streams
URL: https://github.com/apache/pinot/issues/11530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #11530: Protobuf Stream Config validationg assumes Kafka streams
Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #11530:
URL: https://github.com/apache/pinot/issues/11530#issuecomment-1710550794
@xiangfu0 I can create a PR for this. I think @JeffBolle is busy fixing the pulsar connector here - https://github.com/apache/pinot/pull/11542#issuecomment-1710485079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org