You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Gabriel Reid (JIRA)" <ji...@apache.org> on 2012/10/06 14:43:03 UTC
[jira] [Created] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Gabriel Reid created CRUNCH-90:
----------------------------------
Summary: Object reuse is not accounted for in mapper fusion
Key: CRUNCH-90
URL: https://issues.apache.org/jira/browse/CRUNCH-90
Project: Crunch
Issue Type: Bug
Reporter: Gabriel Reid
Assignee: Gabriel Reid
When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471001#comment-13471001 ]
Josh Wills commented on CRUNCH-90:
----------------------------------
+1.
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Attachments: CRUNCH-90.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471333#comment-13471333 ]
Gabriel Reid commented on CRUNCH-90:
------------------------------------
I've just discovered that this patch breaks the PageRankClassTest in scrunch. Unfortunately, I only ran the crunch integration tests (instead of the full suite) before committing it. The exception being thrown is as follows:
java.lang.IllegalArgumentException: Can not set final [Ljava.lang.String; field org.apache.crunch.scrunch.PageRankData.urls to org.apache.avro.generic.GenericData$Array
It appears that the deep copying is running into an issue with reading in a PageRankData object using reflection-based serialization. I'm not sure why this is only coming out now, as I would have thought that existing serialization logic would have caused an issue with it without even using deep copying. This appears to be a bug in Avro (similar to AVRO-1046).
[~jwills] I'm totally clueless when it comes to Scala -- any chance you could take a look and possibly try changing the type of the urls field to something that Avro can deal with, or give me any pointers on what you think might be going on here? I definitely want to report this to the Avro JIRA as well if it is indeed an Avro bug, but it would be good to work around it for now.
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid resolved CRUNCH-90.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Committed
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471596#comment-13471596 ]
Josh Wills commented on CRUNCH-90:
----------------------------------
Most definitely. +1.
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch, CRUNCH-90-reflect.patch, CRUNCH-90-reflect.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471356#comment-13471356 ]
Josh Wills commented on CRUNCH-90:
----------------------------------
Will take a look this evening and see if I can come up with a fix.
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid resolved CRUNCH-90.
--------------------------------
Resolution: Fixed
Committed updated patch
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch, CRUNCH-90-reflect.patch, CRUNCH-90-reflect.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid updated CRUNCH-90:
-------------------------------
Attachment: CRUNCH-90.patch
Patch to fix the issue, as well as tests that demonstrate it. If there are no objections, I'll commit this shortly.
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Attachments: CRUNCH-90.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Gabriel Reid (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabriel Reid updated CRUNCH-90:
-------------------------------
Attachment: CRUNCH-90-reflect.patch
Thanks for figuring that out Josh! And sorry for getting you to do all my monkey work for me, I feel pretty guilty about that now (but it's definitely extra motivation for me to get my act together in terms of figuring Scala out).
I reworked the patch a bit to pass the configuration in via the PType#initialize method instead of PType#getDetachedValue. My dream (or at least my intention) is to have the PType get initialized and automatically made available within DoFns. Sound good to you?
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch, CRUNCH-90-reflect.patch, CRUNCH-90-reflect.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated CRUNCH-90:
-----------------------------
Attachment: CRUNCH-90-reflect.patch
Attached a patch that fixes the PageRankClassTest. In order to do it, I had to besmirch the really elegant DeepCopier infrastructure (like, seriously elegant-- I haven't looked at it that closely before and it was a joy to read) by passing Configuration objects all over the place so that the Avro deepCopy's will use the correct instance of ReflectData-- the Scrunch stuff has to use a different version than the Java stuff.
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch, CRUNCH-90-reflect.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CRUNCH-90) Object reuse is not accounted for in
mapper fusion
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CRUNCH-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills reopened CRUNCH-90:
------------------------------
> Object reuse is not accounted for in mapper fusion
> --------------------------------------------------
>
> Key: CRUNCH-90
> URL: https://issues.apache.org/jira/browse/CRUNCH-90
> Project: Crunch
> Issue Type: Bug
> Reporter: Gabriel Reid
> Assignee: Gabriel Reid
> Fix For: 0.4.0
>
> Attachments: CRUNCH-90.patch
>
>
> When multiple DoFns are run over the same output (i.e. in the case of mapper fusion), the same value object is passed to multiple underlying DoFns. If the state of that value object is changed by one DoFn, other DoFns are called with the updated object.
> This is a situation that can happen quite easily when the input of a DoFn is simply updated and then emitted. In general, this bug will only affect values whose type is the same as the underlying serialization type.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira