You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by srdo <gi...@git.apache.org> on 2018/07/17 15:59:52 UTC
[GitHub] storm pull request #2766: STORM-2972: Replace storm-kafka with storm-kafka-c...
GitHub user srdo opened a pull request:
https://github.com/apache/storm/pull/2766
STORM-2972: Replace storm-kafka with storm-kafka-client in storm-sql-…
…kafka
https://issues.apache.org/jira/browse/STORM-2972
The new kafka scheme syntax is stolen from [Apache Camel](https://github.com/apache/camel/blob/master/components/camel-kafka/src/main/docs/kafka-component.adoc), since I figure we are doing something very similar to them.
The primary limitation here compared to the storm-kafka version is that you can't include metadata (e.g. partition) in the emitted tuple, because I had to adapt the Scheme API to the RecordTranslator API used in storm-kafka-client. Emitting metadata with a Scheme in the storm-kafka spout is handled by [special case instanceof'ing the scheme](https://github.com/apache/storm/blob/af42f434f4a4c3d9087c6058b359033736d3b5e8/external/storm-kafka/src/jvm/org/apache/storm/kafka/KafkaUtils.java#L134) to see if it fits one that takes metadata, and I'd really prefer not to do that.
If we need to be able to emit metadata as well, we can just allow people to override the RecordTranslatorSchemeAdapter. I'll leave that as a task for later, unless someone objects.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srdo/storm STORM-2972
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2766.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2766
----
commit 7f992a6b7474267691e48fd2e5a64230cda41390
Author: Stig Rohde Døssing <sr...@...>
Date: 2018-07-15T15:30:41Z
STORM-2972: Replace storm-kafka with storm-kafka-client in storm-sql-kafka
----
---
[GitHub] storm pull request #2766: STORM-2972: Replace storm-kafka with storm-kafka-c...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2766#discussion_r203588814
--- Diff: docs/storm-sql-example.md ---
@@ -291,9 +291,4 @@ That's it! Supposing we have UDF which queries geo location via remote ip, we ca
## Summary
-We looked through several simple use cases for Storm SQL to learn Storm SQL features. If you haven't looked at [Storm SQL integration](storm-sql.html) and [Storm SQL language](storm-sql-reference.html), you need to read it to see full supported features.
-
-Note that Storm SQL is running on Trident, which is micro-batch, and also no strong typed. Sink doesn't actually check the type.
--- End diff --
Nice finding! Thanks for handling this.
---
[GitHub] storm issue #2766: STORM-2972: Replace storm-kafka with storm-kafka-client i...
Posted by srdo <gi...@git.apache.org>.
Github user srdo commented on the issue:
https://github.com/apache/storm/pull/2766
I tried out storm-sql-kafka with a small example similar to the orders example at https://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-sql.html. I adjusted the docs for the storm-sql-example page, but did not go through and run it.
---
[GitHub] storm pull request #2766: STORM-2972: Replace storm-kafka with storm-kafka-c...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/2766#discussion_r203588887
--- Diff: docs/storm-sql-internal.md ---
@@ -18,18 +18,16 @@ Figure 1 describes the workflow of executing a SQL query in StormSQL. First, use
<p>Figure 1: Workflow of StormSQL.</p>
</div>
-Note: Trident Topology is now replaced with normal Storm topology leveraging Streams API.
--- End diff --
Yeah I forgot to remove the line after renewing diagrams. Nice finding.
---
[GitHub] storm pull request #2766: STORM-2972: Replace storm-kafka with storm-kafka-c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/2766
---