You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by deepak-narkhede <gi...@git.apache.org> on 2017/02/07 11:38:04 UTC
[GitHub] apex-core pull request #466: APEXCORE-634 Apex Platform unable to set unifie...
GitHub user deepak-narkhede opened a pull request:
https://github.com/apache/apex-core/pull/466
APEXCORE-634 Apex Platform unable to set unifier attributes for modules
Problem: Unable to set Unifier attributes of output port within modules
Description: When modules are flatten in logical Plan of DAG, only top level attributes are cloned of OperatorMeta. Unifier attributes are not copied in PortMapping for output ports.
Solution: Clone the unifier attributes while flattening DAG in logical Plan.
Testing done for custom application with and without modules also with app template HDFS to S3 module. Have had debug logs and stack traces to verify the fix.
One of the snip of logs had set the unifier attribute TIMEOUT_WINDOW_COUNT to 10 ( i.e 1000 millisec)
**Without fix:**
**<log_snip>**
2017-02-06 15:51:31,539 WARN com.datatorrent.stram.StreamingContainerManager: UNIFIER operator PTOperator[id=4,name=**genmodule$gen.out#unifier]** committed window ffffffffffffffff, recovery window ffffffffffffffff, current time 1486376491539, last window id change time 0, **window processing timeout millis 60000**
**</log_snip>**
**With Fix:**
**<log_snip>**
2017-02-06 14:22:49,602 WARN com.datatorrent.stram.StreamingContainerManager: UNIFIER operator PTOperator[id=4,name=**genmodule$gen.out#unifier]** committed window ffffffffffffffff, recovery window ffffffffffffffff, current time 1486371169602, last window id change time 0, **window processing timeout millis 1000**
**</log_snip>**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/deepak-narkhede/apex-core APEXCORE-634
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-core/pull/466.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #466
----
commit 3ab8916058e3d5ccaecb9dc84c9a96d6ab22ebec
Author: deepak-narkhede <ma...@gmail.com>
Date: 2017-02-07T11:09:07Z
APEXCORE-634 Apex Platform unable to set unifier attributes for modules in DAG.
Problem: Unable to set Unifier attributes of output port within modules
Description: When modules are flatten in logical Plan of DAG, only top level attributes are cloned of OperatorMeta.
Unifier attributes are not copied in PortMapping for output ports.
Solution: Clone the unifier attributes while flattening DAG in logical Plan.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-core pull request #466: APEXCORE-634 Apex Platform unable to set unifie...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/apex-core/pull/466
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-core pull request #466: APEXCORE-634 Apex Platform unable to set unifie...
Posted by deepak-narkhede <gi...@git.apache.org>.
Github user deepak-narkhede closed the pull request at:
https://github.com/apache/apex-core/pull/466
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-core pull request #466: APEXCORE-634 Apex Platform unable to set unifie...
Posted by deepak-narkhede <gi...@git.apache.org>.
GitHub user deepak-narkhede reopened a pull request:
https://github.com/apache/apex-core/pull/466
APEXCORE-634 Apex Platform unable to set unifier attributes for modules
Problem: Unable to set Unifier attributes of output port within modules
Description: When modules are flatten in logical Plan of DAG, only top level attributes are cloned of OperatorMeta. Unifier attributes are not copied in PortMapping for output ports.
Solution: Clone the unifier attributes while flattening DAG in logical Plan.
Testing done for custom application with and without modules also with app template HDFS to S3 module. Have had debug logs and stack traces to verify the fix.
One of the snip of logs had set the unifier attribute TIMEOUT_WINDOW_COUNT to 10 ( i.e 1000 millisec)
**Without fix:**
**<log_snip>**
2017-02-06 15:51:31,539 WARN com.datatorrent.stram.StreamingContainerManager: UNIFIER operator PTOperator[id=4,name=**genmodule$gen.out#unifier]** committed window ffffffffffffffff, recovery window ffffffffffffffff, current time 1486376491539, last window id change time 0, **window processing timeout millis 60000**
**</log_snip>**
**With Fix:**
**<log_snip>**
2017-02-06 14:22:49,602 WARN com.datatorrent.stram.StreamingContainerManager: UNIFIER operator PTOperator[id=4,name=**genmodule$gen.out#unifier]** committed window ffffffffffffffff, recovery window ffffffffffffffff, current time 1486371169602, last window id change time 0, **window processing timeout millis 1000**
**</log_snip>**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/deepak-narkhede/apex-core APEXCORE-634
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-core/pull/466.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #466
----
commit 12ce7c5d8082289c4690d70b09e65033090c44e1
Author: deepak-narkhede <ma...@gmail.com>
Date: 2017-02-07T11:09:07Z
APEXCORE-634 Apex Platform unable to set unifier attributes for modules in DAG.
Problem: Unable to set Unifier attributes of output port within modules
Description: When modules are flatten in logical Plan of DAG, only top level attributes are cloned of OperatorMeta.
Unifier attributes are not copied in PortMapping for output ports.
Solution: Clone the unifier attributes while flattening DAG in logical Plan.
commit 5df34a81670520cedc74b1ac3f14d1a2d02ca203
Author: deepak-narkhede <ma...@gmail.com>
Date: 2017-02-15T16:09:23Z
APEXCORE-634 Added unit test for testing unifier attribute for module.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---