You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by ilooner <gi...@git.apache.org> on 2016/04/02 01:32:40 UTC
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
GitHub user ilooner opened a pull request:
https://github.com/apache/incubator-apex-core/pull/292
APEXCORE-415 #resolve #comment Fixed double checkpointing bug in Inpu…
…tNode.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilooner/incubator-apex-core APEXCORE-415
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-core/pull/292.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #292
----
commit 5f39cf214126e92ec0d667b66bcae0172c04b259
Author: Timothy Farkas <ti...@datatorrent.com>
Date: 2016-04-01T23:18:41Z
APEXCORE-415 #resolve #comment Fixed double checkpointing bug in InputNode.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205429004
@davidyan74 Will need a lot more refactoring to handle in Node as control tuples are being processed in the specific Node types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205426102
@davidyan74 I think this check happens for other nodes but is not happening for InputNode
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205560349
@ilooner OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-204609434
Travis failure unrelated to my change. All tests pass on my local machine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205426336
@davidyan74 There was a similar bug in Generic Node which I fixed and got merged before. The problem happens when The Operator CheckpointWindowCount is a multiple of the Dag CheckpointWindowCount. In that case the operator is checkpointed on receiving the endWindow tuple and the checkpoint tuple for the same window. When that happens AsyncFSStorageAgent throws an exception because there are two threads trying to move the same file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by sandeshh <gi...@git.apache.org>.
Github user sandeshh commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205428183
How about having the flag in CheckPoint function which decides whether to overwrite the existing checkpoints or not ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205558086
@PramodSSImmaneni @davidyan74 I have added tests. @davidyan74 I don't think your suggestion would work because it does not account for the ProcessMode and doCheckpoint variable interactions. I would like to maintain the same pattern for handling this corner case as existed previously in order to avoid unintended bugs. The code can be refactored down the road after more tests and documentation are introduced.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205426991
@ilooner @PramodSSImmaneni I see. Then shouldn't this change be in Node?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205428578
@ilooner Change looks good, can you add a test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205431850
@PramodSSImmaneni will add test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-apex-core/pull/292
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-204607937
@PramodSSImmaneni please review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by davidyan74 <gi...@git.apache.org>.
Github user davidyan74 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205424902
@ilooner Does this only happen in the InputNode? How about overriding the checkpoint() method in the InputNode for setting and checking for lastCheckpointWindowId without modify the rest of the code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by ilooner <gi...@git.apache.org>.
Github user ilooner commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205413237
@davidyan74 Please review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205432955
@sandeshh as @ilooner mentioned this shouldn't be happening in the first place so I don't think we would need a flag
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-415 #resolve #comment F...
Posted by sandeshh <gi...@git.apache.org>.
Github user sandeshh commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/292#issuecomment-205434534
@PramodSSImmaneni I am for having both the changes in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---