You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/02/06 09:50:23 UTC

[GitHub] [druid] clintropolis opened a new pull request #9320: fix protobuf extension packaging and docs

clintropolis opened a new pull request #9320: fix protobuf extension packaging and docs
URL: https://github.com/apache/druid/pull/9320
 
 
   Fixes #8051.
   
   ### Description
   
   This PR fixes `druid-protobuf-extensions` by tweaking how shading is done to not included the jars that are shaded in the packaging.
   
   before:
   ```
   $ ls -1 apache-druid-0.18.0-SNAPSHOT/extensions/druid-protobuf-extensions/
   druid-protobuf-extensions-0.18.0-SNAPSHOT.jar
   error_prone_annotations-2.3.2.jar
   gson-2.8.6.jar
   protobuf-dynamic-0.9.3.jar
   protobuf-java-3.11.0.jar
   protobuf-java-util-3.11.0.jar
   ```
   
   after:
   ```
   $ ls -1 apache-druid-0.18.0-SNAPSHOT/extensions/druid-protobuf-extensions/
   druid-protobuf-extensions-0.18.0-SNAPSHOT.jar
   ```
   
   I also updated the extension quickstart website docs a bit to make it a bit easier to follow.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added documentation for new or modified features or behaviors.
   - [x] been tested in a test Druid cluster.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] clintropolis commented on issue #9320: fix protobuf extension packaging and docs

Posted by GitBox <gi...@apache.org>.

clintropolis commented on issue #9320: fix protobuf extension packaging and docs
URL: https://github.com/apache/druid/pull/9320#issuecomment-583285989
 
 
   > Doc updates look good. This page doesn't seem to be linked from the left side TOC... is that on purpose?
   
   I believe this is intentional, all of the extensions available are described and linked from http://druid.apache.org/docs/latest/development/extensions.html which is in the 'Configuration' part of the left side TOC.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] clintropolis commented on a change in pull request #9320: fix protobuf extension packaging and docs

Posted by GitBox <gi...@apache.org>.

clintropolis commented on a change in pull request #9320: fix protobuf extension packaging and docs
URL: https://github.com/apache/druid/pull/9320#discussion_r376088414

##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -165,54 +164,74 @@ Please make sure these keys are properly configured for successful ingestion.
}
```

-## Kafka Producer
+## Adding Protobuf messages to Kafka

-Here is the sample script that publishes the metrics to Kafka in Protobuf format.
+If necessary, from your Kafka installation directory run the following command to create the Kafka topic

-1. Run `protoc` again with the Python binding option. This command generates `metrics_pb2.py` file.
- ```
- protoc -o metrics.desc metrics.proto --python_out=.
- ```
+```
+./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic metrics_pb
+```

-2. Create Kafka producer script.
+This example script requires `protobuf` and `kafka-python` modules. With the topic in place, messages can be inserted running the following command from your Druid installation directory

-This script requires `protobuf` and `kafka-python` modules.
+```
+./bin/generate-example-metrics | ./quickstart/protobuf/pb_publisher.py
+```

-```python
-#!/usr/bin/env python
+You can confirm that data has been inserted to your Kafka topic using the following command from your Kafka installation directory
+
+```
+./bin/kafka-console-consumer --zookeeper localhost --topic metrics_pb
+```
+
+which should print messages like this
+
+```
+millisecondsGETR"2017-04-06T03:23:56Z*2002/list:request/latencyBwww1.example.com
+```
+
+If your supervisor created in the previous step is running, the indexing tasks should begin producing the messages and the data will soon be available for querying in Druid.
+
+## Generating the example files
+
+The files provided in the example quickstart can be generated in the following manner starting with only `metrics.proto`.
+
+### `metrics.desc`
+
+The descriptor file is generated using `protoc` Protobuf compiler. Given a `.proto` file, a `.desc` file can be generated like so.
+
+```
+protoc -o metrics.desc metrics.proto
+```
+
+### `metrics_pb2.py`
+`metrics_pb2.py` is also generated with `protoc`
+
+```
+ protoc -o metrics.desc metrics.proto --python_out=.
+```

+### `pb_publisher.py`
+After `metrics_pb2.py` is generated, another script can be constructed to parse JSON data, convert it to Protobuf, and produce to a Kafka topic
+
+```python

Review comment:
Oops, accidentally left that off when i moved stuff around :+1:

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] fjy merged pull request #9320: fix protobuf extension packaging and docs

Posted by GitBox <gi...@apache.org>.

fjy merged pull request #9320: fix protobuf extension packaging and docs
URL: https://github.com/apache/druid/pull/9320
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9320: fix protobuf extension packaging and docs

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9320: fix protobuf extension packaging and docs
URL: https://github.com/apache/druid/pull/9320#discussion_r376033612

##########
File path: docs/development/extensions-core/protobuf.md
##########
@@ -165,54 +164,74 @@ Please make sure these keys are properly configured for successful ingestion.
}
```

-## Kafka Producer
+## Adding Protobuf messages to Kafka

-Here is the sample script that publishes the metrics to Kafka in Protobuf format.
+If necessary, from your Kafka installation directory run the following command to create the Kafka topic

-This script requires `protobuf` and `kafka-python` modules.
+```
+./bin/generate-example-metrics | ./quickstart/protobuf/pb_publisher.py
+```

+### `pb_publisher.py`
+After `metrics_pb2.py` is generated, another script can be constructed to parse JSON data, convert it to Protobuf, and produce to a Kafka topic
+
+```python

Review comment:
Keeping the shebang line that was present before is convenient if this code snippet is copy/pasted into its own script file.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] sthetland commented on issue #9320: fix protobuf extension packaging and docs

Posted by GitBox <gi...@apache.org>.

sthetland commented on issue #9320: fix protobuf extension packaging and docs
URL: https://github.com/apache/druid/pull/9320#issuecomment-583503152
 
 
   > > Doc updates look good. This page doesn't seem to be linked from the left side TOC... is that on purpose?
   > 
   > I believe this is intentional, all of the extensions available are described and linked from http://druid.apache.org/docs/latest/development/extensions.html which is in the 'Configuration' part of the left side TOC.
   
   Got it... that makes sense.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org