You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/05/24 13:01:00 UTC
[jira] [Commented] (METRON-1573) Enhance KAFKA_* functions to
return partition and offset details
[ https://issues.apache.org/jira/browse/METRON-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488944#comment-16488944 ]
ASF GitHub Bot commented on METRON-1573:
----------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1030
METRON-1573 Enhance KAFKA_* functions to return partition and offset details
### Changes
* Updated KAFKA_GET, KAFKA_TAIL, and KAFKA_FIND so that they can each return a "rich" view of messages from Kafka.
* The "rich" view contains not only the message value, but Kafka metadata like the topic, partition, offset, key, and timestamp.
* The change is backwards compatible. By default, only the value is returned just like before.
### Demo
By default, only the message value is returned.
```
[Stellar]>>> KAFKA_GET("bro")
[{"http": {"ts":1527122184.853329,"uid":"CUrRne3iLIxXavQtci","id.orig_h":"192.168.66.1","id.orig_p":50451,"id.resp_h":"192.168.66.121","id.resp_p":8080,"trans_depth":177,"method":"GET","host":"node1","uri":"/api/v1/clusters?fields=Clusters/provisioning_state\u0026_=1484169117876","referrer":"http://node1:8080/","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36","request_body_len":0,"response_body_len":0,"tags":[]}}]
```
Turn on the "rich" view.
```
[Stellar]>>> %define stellar.kafka.message.view := "rich"
rich
```
Fetch the next message.
```
[Stellar]>>> KAFKA_GET("bro")
[{partition=0, offset=2101, topic=bro, value={"http": {"ts":1527122184.674819,"uid":"CUrRne3iLIxXavQtci","id.orig_h":"192.168.66.1","id.orig_p":50451,"id.resp_h":"192.168.66.121","id.resp_p":8080,"trans_depth":24,"method":"GET","host":"node1","uri":"/api/v1/clusters/metron_cluster?fields=Clusters/desired_configs/cluster-env\u0026_=1484168423682","referrer":"http://node1:8080/","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36","request_body_len":0,"response_body_len":0,"tags":[]}}, key=null, timestamp=1527122185741}]
```
Fetch from the tail end of the topic. Notice the offset change.
```
[Stellar]>>> KAFKA_TAIL("bro")
[{partition=0, offset=2600, topic=bro, value={"http": {"ts":1527122376.060672,"uid":"CUrRne3iLIxXavQtci","id.orig_h":"192.168.66.1","id.orig_p":50451,"id.resp_h":"192.168.66.121","id.resp_p":8080,"trans_depth":236,"method":"GET","host":"node1","uri":"/api/v1/clusters/metron_cluster/requests?to=end\u0026page_size=10\u0026fields=Requests\u0026_=1484169369084","referrer":"http://node1:8080/","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36","request_body_len":0,"response_body_len":0,"tags":[]}}, key=null, timestamp=1527122377498}]
```
### Testing
1. Launch the dev environment.
1. Run the Stellar REPL.
1. Try both the "simple" and "rich" views for each of the functions; KAFKA_GET, KAFKA_TAIL and KAFKA_FIND.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1573
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1030.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1030
----
commit 10fe2bfd04db6633b73e9d0f37d458f4ce83aaf9
Author: Nick Allen <ni...@...>
Date: 2018-04-20T20:18:16Z
METRON-1571
commit 73f153692d50af9038dbb9d77412c6298b615efb
Author: Nick Allen <ni...@...>
Date: 2018-05-22T23:17:09Z
METRON-1572 Enhance KAFKA_PUT function
commit 0f06cce8730d2323b65fc114d51ff61ff70b2b8b
Author: Nick Allen <ni...@...>
Date: 2018-05-22T23:43:18Z
METRON-1533 Create KAFKA_FIND Stellar function
commit c7067fe75b7cf9a347dd448062c260a1e9f7f8b4
Author: Nick Allen <ni...@...>
Date: 2018-05-23T00:00:23Z
METRON-1573 Enhance KAFKA_* functions to return partition and offset details
commit 4c4f628750efd23b1564c68ff88459f2b6252fc3
Author: Nick Allen <ni...@...>
Date: 2018-05-23T21:38:12Z
Performing manual partition assignment for KAFKA_GET
commit 4577f401d043ac3072e474201d31fd00be9c9df2
Author: Nick Allen <ni...@...>
Date: 2018-05-24T12:59:58Z
Add property to defaults so it is visible when running KAFKA_PROPS.
----
> Enhance KAFKA_* functions to return partition and offset details
> ----------------------------------------------------------------
>
> Key: METRON-1573
> URL: https://issues.apache.org/jira/browse/METRON-1573
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> The KAFKA_* functions currently simply return the value of the message. There are often times when it would be useful to get more detailed information including the message partition, offset, key, etc.
> The functions should be enhanced to allow a user to optionally get a more detailed view.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)