You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Xinyu Liu <xi...@gmail.com> on 2016/09/16 17:52:22 UTC
Review Request 51962: SAMZA-1021 Remove the redundent poll waiting
inside AsyncRunLoop blockIfBusy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51962/
-----------------------------------------------------------
Review request for samza, Boris Shkolnik, Jake Maes, and Yi Pan (Data Infrastructure).
Repository: samza
Description
-------
The waiting inside AsyncRunLoop is redundent given systemconsumers will block on polling for empty partition. Empty partitions happens when chooser returns null envelope and at the same time some task worker is avaliable to pick up messages. In this case we should rely on the systemconsumers.poll() to block instead of blocking in scheduling.
The redundent waiting has been seen in ATC's ingraph where both choose_ns and block_ns are high at the same time.
Diffs
-----
samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0c5914c772438930d27f100b4d360c1296
Diff: https://reviews.apache.org/r/51962/diff/
Testing
-------
Tested by yarn job.
Thanks,
Xinyu Liu
Re: Review Request 51962: SAMZA-1021 Remove the redundent poll waiting
inside AsyncRunLoop blockIfBusy
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51962/#review149276
-----------------------------------------------------------
Ship it!
Finally found time to understand what you were changing :) yes. We should not block in 2 places for the same cause. +1 !
- Navina Ramesh
On Sept. 16, 2016, 5:56 p.m., Xinyu Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51962/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2016, 5:56 p.m.)
>
>
> Review request for samza, Boris Shkolnik, Jake Maes, and Yi Pan (Data Infrastructure).
>
>
> Bugs: SAMZA-1021
> https://issues.apache.org/jira/browse/SAMZA-1021
>
>
> Repository: samza
>
>
> Description
> -------
>
> The waiting inside AsyncRunLoop is redundent given systemconsumers will block on polling for empty partition. Empty partitions happens when chooser returns null envelope and at the same time some task worker is avaliable to pick up messages. In this case we should rely on the systemconsumers.poll() to block instead of blocking in scheduling.
>
> The redundent waiting has been seen in some our jobs where both choose_ns and block_ns are high at the same time.
>
>
> Diffs
> -----
>
> samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0c5914c772438930d27f100b4d360c1296
>
> Diff: https://reviews.apache.org/r/51962/diff/
>
>
> Testing
> -------
>
> Tested by yarn job.
>
>
> Thanks,
>
> Xinyu Liu
>
>
Re: Review Request 51962: SAMZA-1021 Remove the redundent poll waiting
inside AsyncRunLoop blockIfBusy
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51962/#review149278
-----------------------------------------------------------
Ship it!
Ship It!
- Yi Pan (Data Infrastructure)
On Sept. 16, 2016, 5:56 p.m., Xinyu Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51962/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2016, 5:56 p.m.)
>
>
> Review request for samza, Boris Shkolnik, Jake Maes, and Yi Pan (Data Infrastructure).
>
>
> Bugs: SAMZA-1021
> https://issues.apache.org/jira/browse/SAMZA-1021
>
>
> Repository: samza
>
>
> Description
> -------
>
> The waiting inside AsyncRunLoop is redundent given systemconsumers will block on polling for empty partition. Empty partitions happens when chooser returns null envelope and at the same time some task worker is avaliable to pick up messages. In this case we should rely on the systemconsumers.poll() to block instead of blocking in scheduling.
>
> The redundent waiting has been seen in some our jobs where both choose_ns and block_ns are high at the same time.
>
>
> Diffs
> -----
>
> samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0c5914c772438930d27f100b4d360c1296
>
> Diff: https://reviews.apache.org/r/51962/diff/
>
>
> Testing
> -------
>
> Tested by yarn job.
>
>
> Thanks,
>
> Xinyu Liu
>
>
Re: Review Request 51962: SAMZA-1021 Remove the redundent poll waiting
inside AsyncRunLoop blockIfBusy
Posted by Xinyu Liu <xi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51962/
-----------------------------------------------------------
(Updated Sept. 16, 2016, 5:56 p.m.)
Review request for samza, Boris Shkolnik, Jake Maes, and Yi Pan (Data Infrastructure).
Bugs: SAMZA-1021
https://issues.apache.org/jira/browse/SAMZA-1021
Repository: samza
Description (updated)
-------
The waiting inside AsyncRunLoop is redundent given systemconsumers will block on polling for empty partition. Empty partitions happens when chooser returns null envelope and at the same time some task worker is avaliable to pick up messages. In this case we should rely on the systemconsumers.poll() to block instead of blocking in scheduling.
The redundent waiting has been seen in some our jobs where both choose_ns and block_ns are high at the same time.
Diffs
-----
samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0c5914c772438930d27f100b4d360c1296
Diff: https://reviews.apache.org/r/51962/diff/
Testing
-------
Tested by yarn job.
Thanks,
Xinyu Liu
Re: Review Request 51962: SAMZA-1021 Remove the redundent poll waiting
inside AsyncRunLoop blockIfBusy
Posted by Xinyu Liu <xi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51962/
-----------------------------------------------------------
(Updated Sept. 16, 2016, 5:54 p.m.)
Review request for samza, Boris Shkolnik, Jake Maes, and Yi Pan (Data Infrastructure).
Bugs: SAMZA-1021
https://issues.apache.org/jira/browse/SAMZA-1021
Repository: samza
Description
-------
The waiting inside AsyncRunLoop is redundent given systemconsumers will block on polling for empty partition. Empty partitions happens when chooser returns null envelope and at the same time some task worker is avaliable to pick up messages. In this case we should rely on the systemconsumers.poll() to block instead of blocking in scheduling.
The redundent waiting has been seen in ATC's ingraph where both choose_ns and block_ns are high at the same time.
Diffs
-----
samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0c5914c772438930d27f100b4d360c1296
Diff: https://reviews.apache.org/r/51962/diff/
Testing
-------
Tested by yarn job.
Thanks,
Xinyu Liu
Re: Review Request 51962: SAMZA-1021 Remove the redundent poll waiting
inside AsyncRunLoop blockIfBusy
Posted by Xinyu Liu <xi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51962/
-----------------------------------------------------------
(Updated Sept. 16, 2016, 5:54 p.m.)
Review request for samza, Boris Shkolnik, Jake Maes, and Yi Pan (Data Infrastructure).
Repository: samza
Description
-------
The waiting inside AsyncRunLoop is redundent given systemconsumers will block on polling for empty partition. Empty partitions happens when chooser returns null envelope and at the same time some task worker is avaliable to pick up messages. In this case we should rely on the systemconsumers.poll() to block instead of blocking in scheduling.
The redundent waiting has been seen in ATC's ingraph where both choose_ns and block_ns are high at the same time.
Diffs (updated)
-----
samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0c5914c772438930d27f100b4d360c1296
Diff: https://reviews.apache.org/r/51962/diff/
Testing
-------
Tested by yarn job.
Thanks,
Xinyu Liu