You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by vrozov <gi...@git.apache.org> on 2016/03/15 01:57:17 UTC
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
GitHub user vrozov opened a pull request:
https://github.com/apache/incubator-apex-core/pull/271
APEXCORE-380 - Idle time sleep time should increase from 0 to a configurable max value
@tweise Please review
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vrozov/incubator-apex-core APEXCORE-380
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-core/pull/271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #271
----
commit 08957596ad9e6fb377cdaf61ae1a923ed0453eca
Author: Vlad Rozov <v....@datatorrent.com>
Date: 2016-03-14T21:09:09Z
APEXCORE-380 - Idle time sleep time should increase from 0 to a configurable max value
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by tweise <gi...@git.apache.org>.
Github user tweise commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/271#issuecomment-196593216
@vrozov how much does this change increase the throughput?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by vrozov <gi...@git.apache.org>.
Github user vrozov commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/271#issuecomment-196597969
It depends on the reservoir queue size. For the benchmark application that is configured with 32K queue size, the change leads to 6 million events per second throughput. This is almost 2x improvement compared to 4 million events per second without the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/271#discussion_r56515191
--- Diff: engine/src/main/java/com/datatorrent/stram/engine/GenericNode.java ---
@@ -608,9 +610,9 @@ else if (tracker.ports[trackerIndex] == null) {
if (need2sleep) {
if (handleIdleTime && insideWindow) {
((IdleTimeHandler) operator).handleIdleTime();
- }
- else {
+ } else {
Thread.sleep(spinMillis);
+ spinMillis = Math.min(maxSpinMillis, spinMillis + 1);
--- End diff --
Let me give it a try today just for kicks to see how it performs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by vrozov <gi...@git.apache.org>.
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/271#discussion_r56447191
--- Diff: engine/src/main/java/com/datatorrent/stram/engine/GenericNode.java ---
@@ -608,9 +610,9 @@ else if (tracker.ports[trackerIndex] == null) {
if (need2sleep) {
if (handleIdleTime && insideWindow) {
((IdleTimeHandler) operator).handleIdleTime();
- }
- else {
+ } else {
Thread.sleep(spinMillis);
+ spinMillis = Math.min(maxSpinMillis, spinMillis + 1);
--- End diff --
I also thought about such approach with a variation of providing a configuration setting for a number of iterations instead of a timeout as getting current time on each iteration may be way too costly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by vrozov <gi...@git.apache.org>.
Github user vrozov commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/271#issuecomment-197652278
Another benefit of the current approach is that it is less prone to incorrect sleep time setting. I tried setting sleep time to 100 milliseconds and with the new approach, the throughput is not much affected, with the old approach, the throughput drops significantly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-apex-core/pull/271
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-core pull request: APEXCORE-380 - Idle time sleep t...
Posted by PramodSSImmaneni <gi...@git.apache.org>.
Github user PramodSSImmaneni commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/271#discussion_r56440964
--- Diff: engine/src/main/java/com/datatorrent/stram/engine/GenericNode.java ---
@@ -608,9 +610,9 @@ else if (tracker.ports[trackerIndex] == null) {
if (need2sleep) {
if (handleIdleTime && insideWindow) {
((IdleTimeHandler) operator).handleIdleTime();
- }
- else {
+ } else {
Thread.sleep(spinMillis);
+ spinMillis = Math.min(maxSpinMillis, spinMillis + 1);
--- End diff --
I had looked at this before in a difference scenario. Can we do a better heuristic. We can have another timeout setting (which is less than streaming window size) where we continue to do busy way till that timeout and if there is no data till then, then initiate the stepwise sleep.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---