You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 牛兆捷 <nz...@gmail.com> on 2015/11/04 09:21:36 UTC
Codegen In Shuffle
Dear all:
Tungsten project has mentioned that they are applying code generation is to
speed up the conversion of data from in-memory binary format to
wire-protocol for shuffle.
Where can I find the related implementation in spark code-based ?
--
*Regards,*
*Zhaojie*
Re: Codegen In Shuffle
Posted by 牛兆捷 <nz...@gmail.com>.
I see. Thanks very much.
2015-11-04 16:25 GMT+08:00 Reynold Xin <rx...@databricks.com>:
> GenerateUnsafeProjection -- projects any internal row data structure
> directly into bytes (UnsafeRow).
>
>
> On Wed, Nov 4, 2015 at 12:21 AM, 牛兆捷 <nz...@gmail.com> wrote:
>
>> Dear all:
>>
>> Tungsten project has mentioned that they are applying code generation is
>> to speed up the conversion of data from in-memory binary format to
>> wire-protocol for shuffle.
>>
>> Where can I find the related implementation in spark code-based ?
>>
>> --
>> *Regards,*
>> *Zhaojie*
>>
>>
>
--
*Regards,*
*Zhaojie*
Re: Codegen In Shuffle
Posted by 牛兆捷 <nz...@gmail.com>.
I see. Thanks very much.
2015-11-04 16:25 GMT+08:00 Reynold Xin <rx...@databricks.com>:
> GenerateUnsafeProjection -- projects any internal row data structure
> directly into bytes (UnsafeRow).
>
>
> On Wed, Nov 4, 2015 at 12:21 AM, 牛兆捷 <nz...@gmail.com> wrote:
>
>> Dear all:
>>
>> Tungsten project has mentioned that they are applying code generation is
>> to speed up the conversion of data from in-memory binary format to
>> wire-protocol for shuffle.
>>
>> Where can I find the related implementation in spark code-based ?
>>
>> --
>> *Regards,*
>> *Zhaojie*
>>
>>
>
--
*Regards,*
*Zhaojie*
Re: Codegen In Shuffle
Posted by Reynold Xin <rx...@databricks.com>.
GenerateUnsafeProjection -- projects any internal row data structure
directly into bytes (UnsafeRow).
On Wed, Nov 4, 2015 at 12:21 AM, 牛兆捷 <nz...@gmail.com> wrote:
> Dear all:
>
> Tungsten project has mentioned that they are applying code generation is
> to speed up the conversion of data from in-memory binary format to
> wire-protocol for shuffle.
>
> Where can I find the related implementation in spark code-based ?
>
> --
> *Regards,*
> *Zhaojie*
>
>
Re: Codegen In Shuffle
Posted by Reynold Xin <rx...@databricks.com>.
GenerateUnsafeProjection -- projects any internal row data structure
directly into bytes (UnsafeRow).
On Wed, Nov 4, 2015 at 12:21 AM, 牛兆捷 <nz...@gmail.com> wrote:
> Dear all:
>
> Tungsten project has mentioned that they are applying code generation is
> to speed up the conversion of data from in-memory binary format to
> wire-protocol for shuffle.
>
> Where can I find the related implementation in spark code-based ?
>
> --
> *Regards,*
> *Zhaojie*
>
>