You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/02/20 19:56:40 UTC
[GitHub] [pinot] KKcorps opened a new issue #8234: Dependency isolation in Integration Tests
KKcorps opened a new issue #8234:
URL: https://github.com/apache/pinot/issues/8234
Currently, all of the Pinot integration tests are present in a single module `pinot-integration-tests`. This works fine for most of our modules. But as we are starting to add more and more integrations such as Pulsar, Kinesis, Pub-Sub etc. the current approach leads to a lot of dependency version conflicts. Most of these conflicts occur in libraries such as netty, servlet-api, guava, jackson etc.
In the individual plugins, these dependencies are generally shaded and hence cause no issues when running in production. Resolving each of these dependencies and finding a correct minor version which is compatible with both existing and new plugin takes some effort.
We may need to come up with a new approach.
Some solutions -
* Add integration tests for plugins in their own respective modules only. This will need us to import integration-test modules and other pinot modules in plugins pom (test scope)
* Find a way to minimize depedency conflicts by reducing the number of imported dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps edited a comment on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
KKcorps edited a comment on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1067242215
@Jackie-Jiang Yes solving dependency conflict part among these 3 dependent plugins is fine. But currently, with a common integration-test module, we end up solving conflicts between Kinesis and Kafka as well.
Now, with the approach Xiang mentioned, taking example of your case, we will need to import `pinot-avro` and `pinot-s3` as test dependencies in `pinot-kafka` module. And solving conficts between these would just ruin the `pinot-kafka pom`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 commented on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
xiangfu0 commented on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1046327675
Thanks @KKcorps for bringing this up.
I also have the same feeling that the integration test module has too many dependency conflicts.
I'm +1 on having each plugin module import `pinot-integration-test-base` as test dependency and have the integration test within its own module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1067242215
@Jackie-Jiang Yes solving dependency conflict part among these 3 dependent plugins is fine. But currently, with a common integration-test module, we end up solving conflicts between Kinesis and Kafka as well.
Now, with the approach Xiang mentioned, taking example of your case, we will need to import pinot-avro and pinot-s3 as test dependencies in pinot-kafka module. And solving conficts between these would just ruin the pinot-kafka pom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1064820481
@mayankshriv mentioned a good point. What if an integration test wants to use dependency from other modules? e.g. KafkaIntegrationTest using AvroDecoder?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1065776417
@KKcorps IMO we should make kafka work with avro and S3 because it is a valid setup, but not necessary make kafka, kinesis and pub-sub work with each other as they are serving the same purpose. Essentially we should pick one streaming module, one format module and one deep store module for each integration test module, and solve the dependency conflict between them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1067417458
Actually in order to make a proper distribution with all these modules supported, we need to solve all the conflicts any way. @xiangfu0 How do we make the distribution currently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps edited a comment on issue #8234: Dependency isolation in Integration Tests
Posted by GitBox <gi...@apache.org>.
KKcorps edited a comment on issue #8234:
URL: https://github.com/apache/pinot/issues/8234#issuecomment-1064820481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org