You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Doug Cutting (Created) (JIRA)" <ji...@apache.org> on 2011/10/25 18:34:32 UTC
[jira] [Created] (AVRO-944) Java: reduce-side combine uses wrong
serializer
Java: reduce-side combine uses wrong serializer
-----------------------------------------------
Key: AVRO-944
URL: https://issues.apache.org/jira/browse/AVRO-944
Project: Avro
Issue Type: Bug
Components: java
Reporter: Doug Cutting
Assignee: Doug Cutting
When a reduce-side combine is run, Avro's mapreduce API incorrectly uses the final output serializer rather than the map-output serializer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-944) Java: reduce-side combine uses wrong
serializer
Posted by "Doug Cutting (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-944:
------------------------------
Fix Version/s: (was: 1.6.1)
1.6.0
eclark reported some success with this patch on IRC but intended to do more testing.
I intend to commit this and include it in a new 1.6.0 RC on Monday.
> Java: reduce-side combine uses wrong serializer
> -----------------------------------------------
>
> Key: AVRO-944
> URL: https://issues.apache.org/jira/browse/AVRO-944
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.6.0
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-944.patch
>
>
> When a reduce-side combine is run, Avro's mapreduce API incorrectly uses the final output serializer rather than the map-output serializer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-944) Java: reduce-side combine uses wrong
serializer
Posted by "Julien Muller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139392#comment-13139392 ]
Julien Muller commented on AVRO-944:
------------------------------------
I have jobs that presented the same symptoms when having a combiner.
We ran some testing on our jobs with this patch on 1.5.1 and the result is successful.
This is good news that this patch will be included in the RC, since using the combiner divided the job execution time by 30!
> Java: reduce-side combine uses wrong serializer
> -----------------------------------------------
>
> Key: AVRO-944
> URL: https://issues.apache.org/jira/browse/AVRO-944
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.6.0
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-944.patch
>
>
> When a reduce-side combine is run, Avro's mapreduce API incorrectly uses the final output serializer rather than the map-output serializer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-944) Java: reduce-side combine uses wrong
serializer
Posted by "Doug Cutting (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-944:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I committed this.
> Java: reduce-side combine uses wrong serializer
> -----------------------------------------------
>
> Key: AVRO-944
> URL: https://issues.apache.org/jira/browse/AVRO-944
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.6.0
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.6.0
>
> Attachments: AVRO-944.patch
>
>
> When a reduce-side combine is run, Avro's mapreduce API incorrectly uses the final output serializer rather than the map-output serializer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-944) Java: reduce-side combine uses wrong
serializer
Posted by "Doug Cutting (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-944:
------------------------------
Fix Version/s: 1.6.1
Affects Version/s: 1.6.0
Status: Patch Available (was: Open)
> Java: reduce-side combine uses wrong serializer
> -----------------------------------------------
>
> Key: AVRO-944
> URL: https://issues.apache.org/jira/browse/AVRO-944
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.6.0
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 1.6.1
>
> Attachments: AVRO-944.patch
>
>
> When a reduce-side combine is run, Avro's mapreduce API incorrectly uses the final output serializer rather than the map-output serializer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-944) Java: reduce-side combine uses wrong
serializer
Posted by "Doug Cutting (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-944:
------------------------------
Attachment: AVRO-944.patch
Here's a patch that should fix this. It would be difficult to write a test for this. Can someone who's seen this bug please verify whether this fixes things?
> Java: reduce-side combine uses wrong serializer
> -----------------------------------------------
>
> Key: AVRO-944
> URL: https://issues.apache.org/jira/browse/AVRO-944
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Attachments: AVRO-944.patch
>
>
> When a reduce-side combine is run, Avro's mapreduce API incorrectly uses the final output serializer rather than the map-output serializer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira