You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/11/10 20:59:25 UTC
[GitHub] incubator-metron pull request #354: METRON-557 Create Stellar Functions for ...
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/354
METRON-557 Create Stellar Functions for Kafka
## [METRON-557](https://issues.apache.org/jira/browse/METRON-557)
Add functions to read and write messages to Kafka topics. This should enable a user to debug a Grok expression or Enrichment configuration inside the REPL.
For example, I can pull a message off of the error topic to see why it failed to parse. Then fix the configuration and resubmit the message to try again. I also won\u2019t have to go outside of the Stellar REPL to monitor topic activity.
### `KAFKA_GET`
Retrieves messages from a Kafka topic. Subsequent calls will continue retrieving messages sequentially from the original offset.
Example: Retrieve one message from a topic.
```
KAFKA_GET('topic')
```
Example: Retrieve 10 messages from a topic.
```
KAFKA_GET('topic', 10)
```
Example: Retrieve the first message in a topic. This must be the first retrieval from the topic, otherwise the messages will be retrieved starting from the previously stored consumer offset.
```
KAFKA_GET('topic', 1, { "auto.offset.reset": "earliest" })
```
### `KAFKA_TAIL`
Retrieves messages from a Kafka topic always starting with the most recent message first.
Example: Retrieve the latest message from a topic.
```
KAFKA_TAIL('topic')
```
Example: Retrieve 10 messages from a topic starting with the latest.
```
KAFKA_TAIL('topic', 10)
```
### `KAFKA_PUT`
Sends messages to a Kafka topic.
Example: Put two messages on the topic 'topic'.
```
KAFKA_PUT('topic', ["message1", "message2"])
```
Example: Put a message on a topic and also define an alternative Kafka broker.
```
KAFKA_PUT('topic', ["message1"], { "bootstrap.servers": "kafka-broker-1:6667" })
```
### `KAFKA_PROPS`
Retrieves the Kafka properties that are used by other KAFKA_* functions like KAFKA_GET and KAFKA_PUT. The Kafka properties are compiled from a set of default properties, the global properties, and any overrides.
Example: Retrieve the current Kafka properties.
```
KAFKA_PROPS()
```
Example: Retrieve the current Kafka properties taking into account a set of overrides.
```
KAFKA_PROPS({ "max.poll.records": 1 })
```
## Testing
These were tested on a local Stellar Shell in addition to running within the "Quick Dev" environment. Note that in the "Quick Dev" environment, you must manually copy the jar built within the Management project (`metron-platform/metron-management/target/...`) to `/usr/metron/0.2.1BETA/lib`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-557
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/354.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #354
----
commit 4c2d5daaa603405a0ba8b21b75cd55d8e59efd02
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-11-10T16:32:14Z
METRON-557 Create Stellar Functions for Kafka
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #354: METRON-557 Create Stellar Functions for Kafka
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/354
Yeah, I dig this a lot, @nickwallen +1 by inspection.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #354: METRON-557 Create Stellar Functions for Kafka
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/354
There are functions to read and write from local disk and HDFS [here](https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-management#file-functions). The REPL does allow you to log to disk because it uses a proper GNU readline implementation. You can set it (and a number of other options) by modifying your ~/.inputrc. [Here](http://aeshell.github.io/docs/settings/index.html) are the full options supported.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #354: METRON-557 Create Stellar Functions for Kafka
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/354
Is there something in STELLAR to write to disk?
How about a KAFKA_TEE - output results + log to disk?
Come to think of it, maybe the REPL should have the ability to log everything to disk?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #354: METRON-557 Create Stellar Functions for ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/354
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---