[GitHub] [druid] clintropolis commented on pull request #11126: Add integration test for protobuf

[GitBox <gi...@apache.org>]

clintropolis commented on pull request #11126:
URL: https://github.com/apache/druid/pull/11126#issuecomment-821906937


   Thanks so much for taking this on, this fills one of the last test coverage gaps in the core ingestion extensions!
   
   >Test with schema registry is OK, but it will include kafka-protobuf-provider in pom which is licensed by Confluent Community. Is it OK because is used only in test? If not, how can I do to add it to integration test?
   
   I think we might be able to add the jar(s?) the same way we do for the MySQL connector, which is GPL licensed, by just downloading directly in the Dockerfile, see https://github.com/apache/druid/blob/master/integration-tests/docker/Dockerfile#L45.
   
   >Another problem is file based protobuf encoder needs a file for test, I did't find out how to add it in code, so its test is not work actually. Can you give me some advices to solve them?
   
   Hmm, I think a .proto file with the wikipedia schema could be converted into the .desc file needed for ingestion following the instructions here: https://druid.apache.org/docs/latest/development/extensions-core/protobuf.html#metricsdesc. Then, this descriptor file would need added to git, maybe in https://github.com/apache/druid/tree/master/integration-tests/docker or https://github.com/apache/druid/tree/master/integration-tests/docker/test-data. Next, it would need to be copied to the docker container as part of setup; sample wikipedia data for batch ingestion and some lookup configuration stuff are also copied in a similar manner, see https://github.com/apache/druid/blob/master/integration-tests/script/copy_resources.sh#L81. After this, wherever this file is copied to (path on the container) should now be available to specify in the ingestion spec.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org