[jira] [Commented] (MAPREDUCE-6459) native task crashes when merging spilled file on ppc64

["Tao Jie (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708648#comment-14708648 ] 

Tao Jie commented on MAPREDUCE-6459:
------------------------------------

Char represents [-127, 128] in x86 but [0, 255] in ppc64. 
In atteached patch，when we read or write a char as a integer, we cast the char byte as int8_t which represents [-127, 128] both in x86 and ppc64. 

> native task crashes when merging spilled file on ppc64
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-6459
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6459
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>         Environment: Linux version 2.6.32-431.el6.ppc64
>            Reporter: Tao Jie
>         Attachments: MAPREDUCE-6459.001.patch, ppc64_error.txt
>
>
> when running native task on ppc64，merging spilled files fails since we could not deserialize local spill file correctly.
> Function readVLong in WritableUtils.h and Buffers.h, we try to compare a char with a number and convert a char to int64_t. It does not work correctly on ppc64 since char definition is different between ppc64 and x86 platform. On x86 platform char is defined as signed number while on ppc64 char is unsigned. As a result, we write EOF marker [-1, -1] at the end of spill partition, but deserialize chars as [255, 255].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)