You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by ilganeli <gi...@git.apache.org> on 2016/06/01 23:46:19 UTC
[GitHub] incubator-apex-malhar pull request #307: [APEXMALHAR-2106] Support multiple ...
GitHub user ilganeli opened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/307
[APEXMALHAR-2106] Support multiple streams in StreamMerger
* Added ability to support more than one stream in the operator
Note: I am unclear whether this approach will work if the operator or the DAG restarts, will the ports be reconnected properly? If not, what mechanism prevents this?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilganeli/incubator-apex-malhar APEXMALHAR_2106
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-malhar/pull/307.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #307
----
commit 5a486ff777c3b5c41fa2cbbff7a196bc9da18a44
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2016-06-01T23:36:49Z
Initial implementation of a Stream merger operator that can handle more
than two input streams. Updated test cases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request #307: [APEXMALHAR-2106] Support multiple ...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/307#discussion_r65474494
--- Diff: library/src/main/java/com/datatorrent/lib/stream/StreamMerger.java ---
@@ -41,35 +47,57 @@
@Stateless
public class StreamMerger<K> extends BaseOperator
{
- /**
- * Data input port 1.
- */
- public final transient DefaultInputPort<K> data1 = new DefaultInputPort<K>()
+ private int portCount = 0;
+ private final transient ArrayList<DefaultInputPort<K>> ports = new ArrayList<>(portCount);
--- End diff --
A great question, was in the process of doing so. Short answer is it fails wonderfully:
```
java.lang.IllegalArgumentException: Port is not associated to any operator in the DAG: com.datatorrent.lib.stream.StreamMerger$1@3b084709
at com.datatorrent.stram.plan.logical.LogicalPlan.assertGetPortMeta(LogicalPlan.java:1501)
at com.datatorrent.stram.plan.logical.LogicalPlan.access$1000(LogicalPlan.java:119)
```
The relevant question is thus is there any way to define ports in this way, e.g. an array of ports, such that the DAG is aware of the ports?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request #307: [APEXMALHAR-2106] Support multiple ...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/307#discussion_r65476537
--- Diff: library/src/main/java/com/datatorrent/lib/stream/StreamMerger.java ---
@@ -41,35 +47,57 @@
@Stateless
public class StreamMerger<K> extends BaseOperator
{
- /**
- * Data input port 1.
- */
- public final transient DefaultInputPort<K> data1 = new DefaultInputPort<K>()
+ private int portCount = 0;
+ private final transient ArrayList<DefaultInputPort<K>> ports = new ArrayList<>(portCount);
--- End diff --
No, there isn't. But it is possible to chain multiple mergers with 2 ports and accomplish the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request #307: [APEXMALHAR-2106] Support multiple ...
Posted by gauravgopi123 <gi...@git.apache.org>.
Github user gauravgopi123 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/307#discussion_r65472797
--- Diff: library/src/main/java/com/datatorrent/lib/stream/StreamMerger.java ---
@@ -41,35 +47,57 @@
@Stateless
public class StreamMerger<K> extends BaseOperator
{
- /**
- * Data input port 1.
- */
- public final transient DefaultInputPort<K> data1 = new DefaultInputPort<K>()
+ private int portCount = 0;
+ private final transient ArrayList<DefaultInputPort<K>> ports = new ArrayList<>(portCount);
--- End diff --
Did you use this operator in a DAG and see if it works?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request #307: [APEXMALHAR-2106] Support multiple ...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/307
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---