You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by hmcl <gi...@git.apache.org> on 2017/02/22 17:28:11 UTC
[GitHub] storm pull request #1959: STORM-2374: Storm Kafka Client Test Topologies Mus...
GitHub user hmcl opened a pull request:
https://github.com/apache/storm/pull/1959
STORM-2374: Storm Kafka Client Test Topologies Must be Serializable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hmcl/storm-apache Apache_master_STORM-2374_TpSerial
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1959.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1959
----
commit c2e708b61da1eca183058974ad76e2807c8ab3f6
Author: Hugo Louro <hm...@gmail.com>
Date: 2017-02-22T16:55:12Z
STORM-2374: Storm Kafka Client Test Topologies Must be Serializable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1959
@hmcl
minor: We can remove `Serializable` from RecordTranslator since `Func` already extends it.
With or without the change mentioned above, +1.
I agree that adding Serializable to the existing interface is not safe, but given that `Func` is only used for RecordTranslator, then this approach is easier and clearer since most of implementations of RecordTranslator will have Func as field, and in that case `Func` should be serializable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1959
OK Now I understand what is happening. I think what we really want is to mark Func as Serializable. Could you try the following patch instead?
```
diff --git a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/Func.java b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/Func.java
index a631d9608..041453393 100644
--- a/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/Func.java
+++ b/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/Func.java
@@ -17,10 +17,12 @@
*/
package org.apache.storm.kafka.spout;
+import java.io.Serializable;
+
/**
* A simple interface to allow compatibility with non java 8
* code bases
*/
-public interface Func<V, R> {
+public interface Func<V, R> extends Serializable {
R apply(V record);
}
```
It should work on both master and 1.x
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1959
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1959: STORM-2374: Storm Kafka Client Test Topologies Mus...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1959#discussion_r102533777
--- Diff: external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/test/KafkaSpoutTopologyMainNamedTopics.java ---
@@ -94,10 +98,10 @@ protected StormTopology getTopologyKafkaSpout() {
protected KafkaSpoutConfig<String,String> getKafkaSpoutConfig() {
ByTopicRecordTranslator<String, String> trans = new ByTopicRecordTranslator<>(
- (r) -> new Values(r.topic(), r.partition(), r.offset(), r.key(), r.value()),
--- End diff --
I find it surprising that the lambda function does not work here. I remember explicitly checking to see if it would produce a static lambda or not, and I thought it would create a static one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1959
I am OK with the change but if this is really causing issues then we need to update the docs too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1959
@hmcl This code has not been released yet, or if it was we need to fix this bug in it. This particular Func class was created specifically to make it simple to override specific places in the Kafka spout. They will always need to be serialized out and sent across the wire to the worker, so making it Serializable is a desired thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1959
@revans2 yes, I knew what was the problem. However, it's very strong and dangerous (as for backwards compatibility) to enforce base types (e.g interfaces) to implement `java.io.Serializable`. We do that a lot across the codebase, so I guess it is ok in this case as well.
I was not sure we wanted to make `Func` to extend `Serializable` as it would automatically expose all the internals of all the implementations of this interface. That's the reason I did the proposed fix. If we agree that it's ok to have `Func` extend `Serializable`, I will do as you suggested. Let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the issue:
https://github.com/apache/storm/pull/1959
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1959: STORM-2374: Storm Kafka Client Test Topologies Must be Se...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1959
@revans2 done. @ptgoetz this should be ready to merge. There is also a PR for 1.x-branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1959: STORM-2374: Storm Kafka Client Test Topologies Mus...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1959#discussion_r102604842
--- Diff: external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/test/KafkaSpoutTopologyMainNamedTopics.java ---
@@ -94,10 +98,10 @@ protected StormTopology getTopologyKafkaSpout() {
protected KafkaSpoutConfig<String,String> getKafkaSpoutConfig() {
ByTopicRecordTranslator<String, String> trans = new ByTopicRecordTranslator<>(
- (r) -> new Values(r.topic(), r.partition(), r.offset(), r.key(), r.value()),
--- End diff --
`(Values & Serializable)(r) -> new Values(r.topic(), r.partition(), r.offset(), r.key(), r.value()` might work, but still think Func should be Serializable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1959: STORM-2374: Storm Kafka Client Test Topologies Mus...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1959
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---