You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by greghogan <gi...@git.apache.org> on 2016/06/23 19:25:26 UTC
[GitHub] flink pull request #2156: [FLINK-4113] [runtime] Always copy first value in ...
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/2156
[FLINK-4113] [runtime] Always copy first value in ChainedAllReduceDriver
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink 4113_always_copy_first_value_in_chainedallreducedriver
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2156.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2156
----
commit f5f162b591b38e890758dc65fee93749ac86a347
Author: Greg Hogan <co...@greghogan.com>
Date: 2016-06-23T16:37:37Z
[FLINK-4113] [runtime] Always copy first value in ChainedAllReduceDriver
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2156: [FLINK-4113] [runtime] Always copy first value in ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2156
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2156: [FLINK-4113] [runtime] Always copy first value in Chained...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2156
@greghogan the tests looks good. Thanks for your work :-) The failing test cases are unrelated. Will merge this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2156: [FLINK-4113] [runtime] Always copy first value in ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2156#discussion_r68560241
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllReduceDriver.java ---
@@ -89,7 +89,7 @@ public void collect(IT record) {
numRecordsIn.inc();
try {
if (base == null) {
- base = objectReuseEnabled ? record : serializer.copy(record);
+ base = serializer.copy(record);
} else {
base = objectReuseEnabled ? reducer.reduce(base, record) : serializer.copy(reducer.reduce(base, record));
--- End diff --
What if we return the `record` as the result of `reducer.reduce` (with `objectReuseEnabled == true`)? That would also be problematic, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2156: [FLINK-4113] [runtime] Always copy first value in ...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/2156#discussion_r68566231
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllReduceDriver.java ---
@@ -89,7 +89,7 @@ public void collect(IT record) {
numRecordsIn.inc();
try {
if (base == null) {
- base = objectReuseEnabled ? record : serializer.copy(record);
+ base = serializer.copy(record);
} else {
base = objectReuseEnabled ? reducer.reduce(base, record) : serializer.copy(reducer.reduce(base, record));
--- End diff --
We fixed this in FLINK-3340 for non-chained reduce drivers (where the driver chooses the object to deserialize into) but for chained drivers we cannot prevent one UDF from overwriting an object from a previous UDF. If you look in {{OverwriteObjects.java}} you will see {{testReduce}} fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2156: [FLINK-4113] [runtime] Always copy first value in Chained...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2156
Good fix @greghogan. We should definitely include it in the upcoming release. Would it be possible to write a test to guard the fix from future changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2156: [FLINK-4113] [runtime] Always copy first value in Chained...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2156
@tillrohrmann just added a test for `ChainedAllReduceDriver`. I didn't see a way to build a lightweight test for chained drivers as we do with the unchained drivers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2156: [FLINK-4113] [runtime] Always copy first value in ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2156#discussion_r68718332
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedAllReduceDriver.java ---
@@ -89,7 +89,7 @@ public void collect(IT record) {
numRecordsIn.inc();
try {
if (base == null) {
- base = objectReuseEnabled ? record : serializer.copy(record);
+ base = serializer.copy(record);
} else {
base = objectReuseEnabled ? reducer.reduce(base, record) : serializer.copy(reducer.reduce(base, record));
--- End diff --
Alright. I guess users then have to live with that if they enable object reuse.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---