You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Lu Niu <qq...@gmail.com> on 2023/04/26 17:33:23 UTC

flink batch execution mode

Hi, Flink users

I am trying to understand the internals of flink batch mode. some questions:

1. Does flink batch mode use columnar in-memory format?
2. Does flink batch mode use vectorization technique?
3. any performance benchmark available compared with batch engines like
spark or presto?

Some code pointers would be great! Thanks!

Best
Lu

Re: flink batch execution mode

Posted by Lu Niu <qq...@gmail.com>.

Thanks! Shammon! Do you also have insights on the first 2 questions as
well? Thanks!

1. Does flink batch mode use columnar in-memory format?
2. Does flink batch mode use vectorization technique?

Best
Lu


On Wed, Apr 26, 2023 at 7:51 PM Shammon FY <zj...@gmail.com> wrote:

> Hi Lu,
>
> Currently, Flink does not have official benchmark results compared to
> Spark and Presto. You can run the TPC-DS benchmark yourself to compare to
> different engines, Flink supports all queries for TPC-DS, for example, the
> benchmark project [1]. Of course, other companies have made similar
> comparisons, such as this document [2], and the results are for reference
> only.
>
> [1] https://github.com/ververica/flink-sql-benchmark
> [2]
> https://files.alicdn.com/tpsservice/c399186c83bb17dc5dd72cbd0c6ccf71.pdf?spm=a2csy.flink.0.0.49496ea8lXSLb8&file=c399186c83bb17dc5dd72cbd0c6ccf71.pdf
>
> Best,
> Shammon FY
>
> On Thu, Apr 27, 2023 at 1:33 AM Lu Niu <qq...@gmail.com> wrote:
>
>> Hi, Flink users
>>
>> I am trying to understand the internals of flink batch mode. some
>> questions:
>>
>> 1. Does flink batch mode use columnar in-memory format?
>> 2. Does flink batch mode use vectorization technique?
>> 3. any performance benchmark available compared with batch engines like
>> spark or presto?
>>
>> Some code pointers would be great! Thanks!
>>
>> Best
>> Lu
>>
>>
>>

Re: flink batch execution mode

Posted by weijie guo <gu...@gmail.com>.

Hi Lu,

At present, Flink is still based on row format in the runtime execution
layer and vectorization relies mostly on automatic compiler optimizations
of the JVM.


Best regards,

Weijie


Shammon FY <zj...@gmail.com> 于2023年4月27日周四 10:52写道：

> Hi Lu,
>
> Currently, Flink does not have official benchmark results compared to
> Spark and Presto. You can run the TPC-DS benchmark yourself to compare to
> different engines, Flink supports all queries for TPC-DS, for example, the
> benchmark project [1]. Of course, other companies have made similar
> comparisons, such as this document [2], and the results are for reference
> only.
>
> [1] https://github.com/ververica/flink-sql-benchmark
> [2]
> https://files.alicdn.com/tpsservice/c399186c83bb17dc5dd72cbd0c6ccf71.pdf?spm=a2csy.flink.0.0.49496ea8lXSLb8&file=c399186c83bb17dc5dd72cbd0c6ccf71.pdf
>
> Best,
> Shammon FY
>
> On Thu, Apr 27, 2023 at 1:33 AM Lu Niu <qq...@gmail.com> wrote:
>
>> Hi, Flink users
>>
>> I am trying to understand the internals of flink batch mode. some
>> questions:
>>
>> 1. Does flink batch mode use columnar in-memory format?
>> 2. Does flink batch mode use vectorization technique?
>> 3. any performance benchmark available compared with batch engines like
>> spark or presto?
>>
>> Some code pointers would be great! Thanks!
>>
>> Best
>> Lu
>>
>>
>>

Re: flink batch execution mode

Posted by Shammon FY <zj...@gmail.com>.

Hi Lu,

Currently, Flink does not have official benchmark results compared to Spark
and Presto. You can run the TPC-DS benchmark yourself to compare to
different engines, Flink supports all queries for TPC-DS, for example, the
benchmark project [1]. Of course, other companies have made similar
comparisons, such as this document [2], and the results are for reference
only.

[1] https://github.com/ververica/flink-sql-benchmark
[2]
https://files.alicdn.com/tpsservice/c399186c83bb17dc5dd72cbd0c6ccf71.pdf?spm=a2csy.flink.0.0.49496ea8lXSLb8&file=c399186c83bb17dc5dd72cbd0c6ccf71.pdf

Best,
Shammon FY

On Thu, Apr 27, 2023 at 1:33 AM Lu Niu <qq...@gmail.com> wrote:

> Hi, Flink users
>
> I am trying to understand the internals of flink batch mode. some
> questions:
>
> 1. Does flink batch mode use columnar in-memory format?
> 2. Does flink batch mode use vectorization technique?
> 3. any performance benchmark available compared with batch engines like
> spark or presto?
>
> Some code pointers would be great! Thanks!
>
> Best
> Lu
>
>
>