You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by ilooner <gi...@git.apache.org> on 2015/11/13 22:20:23 UTC
[GitHub] incubator-apex-core pull request: - APEX-263 Fixed case where doub...
GitHub user ilooner opened a pull request:
https://github.com/apache/incubator-apex-core/pull/167
- APEX-263 Fixed case where double checkpointing could occurr.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilooner/incubator-apex-core APEX-263
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-core/pull/167.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #167
----
commit 8bd67d6c85cf043fd511051985073696931837d7
Author: Timothy Farkas <ti...@datatorrent.com>
Date: 2015-11-13T20:20:33Z
- APEX-263 Fixed case where double checkpointing could occurr.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: - APEX-263 Fixed case where doub...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/167#discussion_r44957610
--- Diff: engine/src/test/java/com/datatorrent/stram/engine/GenericNodeTest.java ---
@@ -296,4 +323,67 @@ public void run()
Assert.assertTrue("End window not called", go.endWindowId != go.beginWindowId);
}
+ @Test
+ public void testDoubleCheckpointAtleastOnce() throws Exception
+ {
+ testDoubleCheckpointHandling(ProcessingMode.AT_LEAST_ONCE);
+ }
+
+ @Test
+ public void testDoubleCheckpointAtMostOnce() throws Exception
+ {
+ testDoubleCheckpointHandling(ProcessingMode.AT_MOST_ONCE);
+ }
+
+ @Test
+ public void testDoubleCheckpointExactlyOnce() throws Exception
+ {
+ testDoubleCheckpointHandling(ProcessingMode.EXACTLY_ONCE);
+ }
+
+ private void testDoubleCheckpointHandling(ProcessingMode processingMode) throws Exception
+ {
+ WindowGenerator windowGenerator = new WindowGenerator(new ScheduledThreadPoolExecutor(1, "WindowGenerator"), 1024);
+ windowGenerator.setResetWindow(0L);
+ windowGenerator.setFirstWindow(0L);
+ windowGenerator.setWindowWidth(100);
+ windowGenerator.setCheckpointCount(1, 0);
+
+ GenericCheckpointOperator gco = new GenericCheckpointOperator();
+ DefaultAttributeMap dam = new DefaultAttributeMap();
+ dam.put(OperatorContext.APPLICATION_WINDOW_COUNT, 2);
+ dam.put(OperatorContext.CHECKPOINT_WINDOW_COUNT, 2);
+
+ final GenericNode in = new GenericNode(gco, new com.datatorrent.stram.engine.OperatorContext(0, dam, null));
+ in.setId(1);
+
+ TestSink testSink = new TestSink();
+
+ in.connectInputPort("ip1", windowGenerator.acquireReservoir(String.valueOf(in.id), 1024));
+ in.connectOutputPort("output", testSink);
+
+ windowGenerator.activate(null);
+
+ final AtomicBoolean ab = new AtomicBoolean(false);
+ Thread t = new Thread()
+ {
+ @Override
+ public void run()
+ {
+ ab.set(true);
+ in.activate();
+ in.run();
+ in.deactivate();
+ }
+
+ };
+
+ t.start();
+ Thread.sleep(3000);
--- End diff --
Wait till 3 windows instead of fixed timeout
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: - APEX-263 Fixed case where doub...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/167#discussion_r44966554
--- Diff: engine/src/test/java/com/datatorrent/stram/engine/GenericNodeTest.java ---
@@ -296,4 +323,67 @@ public void run()
Assert.assertTrue("End window not called", go.endWindowId != go.beginWindowId);
}
+ @Test
+ public void testDoubleCheckpointAtleastOnce() throws Exception
+ {
+ testDoubleCheckpointHandling(ProcessingMode.AT_LEAST_ONCE);
+ }
+
+ @Test
+ public void testDoubleCheckpointAtMostOnce() throws Exception
+ {
+ testDoubleCheckpointHandling(ProcessingMode.AT_MOST_ONCE);
+ }
+
+ @Test
+ public void testDoubleCheckpointExactlyOnce() throws Exception
+ {
+ testDoubleCheckpointHandling(ProcessingMode.EXACTLY_ONCE);
+ }
+
+ private void testDoubleCheckpointHandling(ProcessingMode processingMode) throws Exception
+ {
+ WindowGenerator windowGenerator = new WindowGenerator(new ScheduledThreadPoolExecutor(1, "WindowGenerator"), 1024);
+ windowGenerator.setResetWindow(0L);
+ windowGenerator.setFirstWindow(0L);
+ windowGenerator.setWindowWidth(100);
+ windowGenerator.setCheckpointCount(1, 0);
+
+ GenericCheckpointOperator gco = new GenericCheckpointOperator();
+ DefaultAttributeMap dam = new DefaultAttributeMap();
+ dam.put(OperatorContext.APPLICATION_WINDOW_COUNT, 2);
+ dam.put(OperatorContext.CHECKPOINT_WINDOW_COUNT, 2);
+
+ final GenericNode in = new GenericNode(gco, new com.datatorrent.stram.engine.OperatorContext(0, dam, null));
+ in.setId(1);
+
+ TestSink testSink = new TestSink();
+
+ in.connectInputPort("ip1", windowGenerator.acquireReservoir(String.valueOf(in.id), 1024));
+ in.connectOutputPort("output", testSink);
+
+ windowGenerator.activate(null);
+
+ final AtomicBoolean ab = new AtomicBoolean(false);
+ Thread t = new Thread()
+ {
+ @Override
+ public void run()
+ {
+ ab.set(true);
+ in.activate();
+ in.run();
+ in.deactivate();
+ }
+
+ };
+
+ t.start();
+ Thread.sleep(3000);
--- End diff --
@PramodSSImmaneni made the change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-apex-core pull request: - APEX-263 Fixed case
where doub...
Posted by "Chetan Narsude (cnarsude)" <cn...@cisco.com>.
You need one on line:164 as well.
‹
Chetan
On 11/13/15, 1:21 PM, "ilooner" <gi...@git.apache.org> wrote:
>Github user ilooner commented on the pull request:
>
>
>https://github.com/apache/incubator-apex-core/pull/167#issuecomment-156559
>640
>
> This is for review only @PramodSSImmaneni and @243826 Please this is
>a small fix. I will work on writing a unit test in the mean time.
>
>
>---
>If your project is set up for it, you can reply to this email and have
>your
>reply appear on GitHub as well. If your project does not have this feature
>enabled and wishes so, or if the feature is enabled but not working,
>please
>contact infrastructure at infrastructure@apache.org or file a JIRA ticket
>with INFRA.
>---
[GitHub] incubator-apex-core pull request: - APEX-263 Fixed case where doub...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/167#issuecomment-156559640
This is for review only @PramodSSImmaneni and @243826 Please this is a small fix. I will work on writing a unit test in the mean time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: - APEX-263 Fixed case where doub...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/167#issuecomment-156775334
@PramodSSImmaneni @243826 fixed exactly once case and added unit tests. Please review I think it is ready to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: - APEX-263 Fixed case where doub...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-apex-core/pull/167
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---