You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jacky Lau <li...@gmail.com> on 2026/10/15 03:13:54 UTC

[DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Hi guys:
     I want to introduce all of you a flink naive engine StreamFusion which
is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It has
good performance and supports lots of sql operator.
    Will it be move it to flink subproject like
flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
Query Memory, off Memory, on Disk, off Disk, on
q0 1.69× 1.39× 1.36× 1.40×
q1† 1.40× 1.40× 1.38× 1.41×
q2 1.26× 1.08× 1.11× 1.08×
q3 1.03× 1.16× 1.08× 1.14×
q4 1.83× 1.60× 8.92× 15.11×
q5 1.41× 1.23× 4.38× 4.38×
q7 1.30× 1.47× 7.41× 11.93×
q8 1.27× 1.12× 2.33× 2.43×
q9 1.22× 1.80× 17.83× 67.95×
q10† 1.45× 1.94× 1.42× 1.40×
q11 1.47× 1.50× 9.84× 10.03×
q12 1.09× 1.13× 1.45× 1.33×
q13 1.20× 1.14× 1.15× 1.22×
q14† 1.47× 1.25× 1.49× 1.53×
q15† 1.47× 1.22× 1.61× 1.32×
q16† 1.17× 1.41× 1.49× 1.28×
q17† 1.20× 1.18× 1.92× 1.65×
q18 1.07× 1.37× 6.01× 8.92×
q19 0.99× 2.82× 2.44× 5.09×
q20 1.22× 1.56× 37.43× 68.20×
q21† 1.23× 1.31× 1.26× 1.27×
q22 1.33× 1.25× 1.30× 1.22×
q23 1.69× 3.32× 3.15× 12.10×
geomean 1.31× 1.44× 2.75× 3.41×

† The benchmark's existing opt-in expression variants are used for these
headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
q21 use native datetime or regex/case behavior that can differ from Flink
at documented edge cases.

[1] https://github.com/datafusion-contrib/StreamFusion
[2] https://streamfusion.tech/ .

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Yong Liu <li...@gmail.com>.
Thanks for your join this discussion. Looking forward to your progress.

Yongming Zhang <da...@gmail.com> 于2026年10月15日周四 11:46写道:

> I also discovered this project recently and a quick performance test
> showed excellent results, and the codebase is very well-developed. My
> company is currently in the process of integrating and evaluating it. I
> think incorporating it as a sub-project of Flink is a fantastic idea—it
> would also help further enrich the Flink community ecosystem.
>
> Jacky Lau <li...@gmail.com> 于2026年10月15日周四 11:14写道:
>
>> Hi guys:
>>      I want to introduce all of you a flink naive engine StreamFusion
>> which is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It
>> has good performance and supports lots of sql operator.
>>     Will it be move it to flink subproject like
>> flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
>> Query Memory, off Memory, on Disk, off Disk, on
>> q0 1.69× 1.39× 1.36× 1.40×
>> q1† 1.40× 1.40× 1.38× 1.41×
>> q2 1.26× 1.08× 1.11× 1.08×
>> q3 1.03× 1.16× 1.08× 1.14×
>> q4 1.83× 1.60× 8.92× 15.11×
>> q5 1.41× 1.23× 4.38× 4.38×
>> q7 1.30× 1.47× 7.41× 11.93×
>> q8 1.27× 1.12× 2.33× 2.43×
>> q9 1.22× 1.80× 17.83× 67.95×
>> q10† 1.45× 1.94× 1.42× 1.40×
>> q11 1.47× 1.50× 9.84× 10.03×
>> q12 1.09× 1.13× 1.45× 1.33×
>> q13 1.20× 1.14× 1.15× 1.22×
>> q14† 1.47× 1.25× 1.49× 1.53×
>> q15† 1.47× 1.22× 1.61× 1.32×
>> q16† 1.17× 1.41× 1.49× 1.28×
>> q17† 1.20× 1.18× 1.92× 1.65×
>> q18 1.07× 1.37× 6.01× 8.92×
>> q19 0.99× 2.82× 2.44× 5.09×
>> q20 1.22× 1.56× 37.43× 68.20×
>> q21† 1.23× 1.31× 1.26× 1.27×
>> q22 1.33× 1.25× 1.30× 1.22×
>> q23 1.69× 3.32× 3.15× 12.10×
>> geomean 1.31× 1.44× 2.75× 3.41×
>>
>> † The benchmark's existing opt-in expression variants are used for these
>> headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
>> q21 use native datetime or regex/case behavior that can differ from Flink
>> at documented edge cases.
>>
>> [1] https://github.com/datafusion-contrib/StreamFusion
>> [2] https://streamfusion.tech/ .
>>
>

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Yong Liu <li...@gmail.com>.
Thanks for your join this discussion. Looking forward to your progress.

Yongming Zhang <da...@gmail.com> 于2026年10月15日周四 11:46写道:

> I also discovered this project recently and a quick performance test
> showed excellent results, and the codebase is very well-developed. My
> company is currently in the process of integrating and evaluating it. I
> think incorporating it as a sub-project of Flink is a fantastic idea—it
> would also help further enrich the Flink community ecosystem.
>
> Jacky Lau <li...@gmail.com> 于2026年10月15日周四 11:14写道:
>
>> Hi guys:
>>      I want to introduce all of you a flink naive engine StreamFusion
>> which is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It
>> has good performance and supports lots of sql operator.
>>     Will it be move it to flink subproject like
>> flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
>> Query Memory, off Memory, on Disk, off Disk, on
>> q0 1.69× 1.39× 1.36× 1.40×
>> q1† 1.40× 1.40× 1.38× 1.41×
>> q2 1.26× 1.08× 1.11× 1.08×
>> q3 1.03× 1.16× 1.08× 1.14×
>> q4 1.83× 1.60× 8.92× 15.11×
>> q5 1.41× 1.23× 4.38× 4.38×
>> q7 1.30× 1.47× 7.41× 11.93×
>> q8 1.27× 1.12× 2.33× 2.43×
>> q9 1.22× 1.80× 17.83× 67.95×
>> q10† 1.45× 1.94× 1.42× 1.40×
>> q11 1.47× 1.50× 9.84× 10.03×
>> q12 1.09× 1.13× 1.45× 1.33×
>> q13 1.20× 1.14× 1.15× 1.22×
>> q14† 1.47× 1.25× 1.49× 1.53×
>> q15† 1.47× 1.22× 1.61× 1.32×
>> q16† 1.17× 1.41× 1.49× 1.28×
>> q17† 1.20× 1.18× 1.92× 1.65×
>> q18 1.07× 1.37× 6.01× 8.92×
>> q19 0.99× 2.82× 2.44× 5.09×
>> q20 1.22× 1.56× 37.43× 68.20×
>> q21† 1.23× 1.31× 1.26× 1.27×
>> q22 1.33× 1.25× 1.30× 1.22×
>> q23 1.69× 3.32× 3.15× 12.10×
>> geomean 1.31× 1.44× 2.75× 3.41×
>>
>> † The benchmark's existing opt-in expression variants are used for these
>> headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
>> q21 use native datetime or regex/case behavior that can differ from Flink
>> at documented edge cases.
>>
>> [1] https://github.com/datafusion-contrib/StreamFusion
>> [2] https://streamfusion.tech/ .
>>
>

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Yongming Zhang <da...@gmail.com>.
I also discovered this project recently and a quick performance test showed
excellent results, and the codebase is very well-developed. My company is
currently in the process of integrating and evaluating it. I think
incorporating it as a sub-project of Flink is a fantastic idea—it would
also help further enrich the Flink community ecosystem.

Jacky Lau <li...@gmail.com> 于2026年10月15日周四 11:14写道:

> Hi guys:
>      I want to introduce all of you a flink naive engine StreamFusion
> which is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It
> has good performance and supports lots of sql operator.
>     Will it be move it to flink subproject like
> flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
> Query Memory, off Memory, on Disk, off Disk, on
> q0 1.69× 1.39× 1.36× 1.40×
> q1† 1.40× 1.40× 1.38× 1.41×
> q2 1.26× 1.08× 1.11× 1.08×
> q3 1.03× 1.16× 1.08× 1.14×
> q4 1.83× 1.60× 8.92× 15.11×
> q5 1.41× 1.23× 4.38× 4.38×
> q7 1.30× 1.47× 7.41× 11.93×
> q8 1.27× 1.12× 2.33× 2.43×
> q9 1.22× 1.80× 17.83× 67.95×
> q10† 1.45× 1.94× 1.42× 1.40×
> q11 1.47× 1.50× 9.84× 10.03×
> q12 1.09× 1.13× 1.45× 1.33×
> q13 1.20× 1.14× 1.15× 1.22×
> q14† 1.47× 1.25× 1.49× 1.53×
> q15† 1.47× 1.22× 1.61× 1.32×
> q16† 1.17× 1.41× 1.49× 1.28×
> q17† 1.20× 1.18× 1.92× 1.65×
> q18 1.07× 1.37× 6.01× 8.92×
> q19 0.99× 2.82× 2.44× 5.09×
> q20 1.22× 1.56× 37.43× 68.20×
> q21† 1.23× 1.31× 1.26× 1.27×
> q22 1.33× 1.25× 1.30× 1.22×
> q23 1.69× 3.32× 3.15× 12.10×
> geomean 1.31× 1.44× 2.75× 3.41×
>
> † The benchmark's existing opt-in expression variants are used for these
> headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
> q21 use native datetime or regex/case behavior that can differ from Flink
> at documented edge cases.
>
> [1] https://github.com/datafusion-contrib/StreamFusion
> [2] https://streamfusion.tech/ .
>

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Jacky Lau <li...@gmail.com>.
StreamFusion doesn't change any flink code ,just move the jar to the lib
and add a configuration flag to enable this feature. and after dig the
details  of it, the project current design is very good.so i advice put it
as sub project.
1. for the planner: StreamFusion just by matches to  replace Physical Plan
and ExecPlan to implement Native Operator replace
2. for the runtime: it reuse the checkpointing and tolerance.
3. for the stateful operator: it refers risingwave/arroyo some design and
use multiget/multiset to optimize the i/o performance
4. for production: the development is very quickly by AI. and feature/issue
will be supported quickly.

Ran Tao <ch...@gmail.com> 于2026年10月15日周四 15:18写道:

> Hi Jacky,
>
> Thanks for sharing StreamFusion! This looks like an interesting project.
>
> I'm curious about how StreamFusion integrates with Flink's existing planner
> and runtime, especially regarding stateful streaming, checkpointing, and
> fault tolerance.
>
> Also, has StreamFusion been used in production yet? It would be great to
> hear about any real use cases and experiences.
>
> Best Regards,
> Ran Tao
> https://github.com/taoran92
>
>
> Jacky Lau <li...@gmail.com> 于2026年10月15日周四 11:14写道:
>
> > Hi guys:
> >      I want to introduce all of you a flink naive engine StreamFusion
> which
> > is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It has
> > good performance and supports lots of sql operator.
> >     Will it be move it to flink subproject like
> > flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
> > Query Memory, off Memory, on Disk, off Disk, on
> > q0 1.69× 1.39× 1.36× 1.40×
> > q1† 1.40× 1.40× 1.38× 1.41×
> > q2 1.26× 1.08× 1.11× 1.08×
> > q3 1.03× 1.16× 1.08× 1.14×
> > q4 1.83× 1.60× 8.92× 15.11×
> > q5 1.41× 1.23× 4.38× 4.38×
> > q7 1.30× 1.47× 7.41× 11.93×
> > q8 1.27× 1.12× 2.33× 2.43×
> > q9 1.22× 1.80× 17.83× 67.95×
> > q10† 1.45× 1.94× 1.42× 1.40×
> > q11 1.47× 1.50× 9.84× 10.03×
> > q12 1.09× 1.13× 1.45× 1.33×
> > q13 1.20× 1.14× 1.15× 1.22×
> > q14† 1.47× 1.25× 1.49× 1.53×
> > q15† 1.47× 1.22× 1.61× 1.32×
> > q16† 1.17× 1.41× 1.49× 1.28×
> > q17† 1.20× 1.18× 1.92× 1.65×
> > q18 1.07× 1.37× 6.01× 8.92×
> > q19 0.99× 2.82× 2.44× 5.09×
> > q20 1.22× 1.56× 37.43× 68.20×
> > q21† 1.23× 1.31× 1.26× 1.27×
> > q22 1.33× 1.25× 1.30× 1.22×
> > q23 1.69× 3.32× 3.15× 12.10×
> > geomean 1.31× 1.44× 2.75× 3.41×
> >
> > † The benchmark's existing opt-in expression variants are used for these
> > headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
> > q21 use native datetime or regex/case behavior that can differ from Flink
> > at documented edge cases.
> >
> > [1] https://github.com/datafusion-contrib/StreamFusion
> > [2] https://streamfusion.tech/ .
> >
>

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Jacky Lau <li...@gmail.com>.
StreamFusion doesn't change any flink code ,just move the jar to the lib
and add a configuration flag to enable this feature. and after dig the
details  of it, the project current design is very good.so i advice put it
as sub project.
1. for the planner: StreamFusion just by matches to  replace Physical Plan
and ExecPlan to implement Native Operator replace
2. for the runtime: it reuse the checkpointing and tolerance.
3. for the stateful operator: it refers risingwave/arroyo some design and
use multiget/multiset to optimize the i/o performance
4. for production: the development is very quickly by AI. and feature/issue
will be supported quickly.

Ran Tao <ch...@gmail.com> 于2026年10月15日周四 15:18写道:

> Hi Jacky,
>
> Thanks for sharing StreamFusion! This looks like an interesting project.
>
> I'm curious about how StreamFusion integrates with Flink's existing planner
> and runtime, especially regarding stateful streaming, checkpointing, and
> fault tolerance.
>
> Also, has StreamFusion been used in production yet? It would be great to
> hear about any real use cases and experiences.
>
> Best Regards,
> Ran Tao
> https://github.com/taoran92
>
>
> Jacky Lau <li...@gmail.com> 于2026年10月15日周四 11:14写道:
>
> > Hi guys:
> >      I want to introduce all of you a flink naive engine StreamFusion
> which
> > is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It has
> > good performance and supports lots of sql operator.
> >     Will it be move it to flink subproject like
> > flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
> > Query Memory, off Memory, on Disk, off Disk, on
> > q0 1.69× 1.39× 1.36× 1.40×
> > q1† 1.40× 1.40× 1.38× 1.41×
> > q2 1.26× 1.08× 1.11× 1.08×
> > q3 1.03× 1.16× 1.08× 1.14×
> > q4 1.83× 1.60× 8.92× 15.11×
> > q5 1.41× 1.23× 4.38× 4.38×
> > q7 1.30× 1.47× 7.41× 11.93×
> > q8 1.27× 1.12× 2.33× 2.43×
> > q9 1.22× 1.80× 17.83× 67.95×
> > q10† 1.45× 1.94× 1.42× 1.40×
> > q11 1.47× 1.50× 9.84× 10.03×
> > q12 1.09× 1.13× 1.45× 1.33×
> > q13 1.20× 1.14× 1.15× 1.22×
> > q14† 1.47× 1.25× 1.49× 1.53×
> > q15† 1.47× 1.22× 1.61× 1.32×
> > q16† 1.17× 1.41× 1.49× 1.28×
> > q17† 1.20× 1.18× 1.92× 1.65×
> > q18 1.07× 1.37× 6.01× 8.92×
> > q19 0.99× 2.82× 2.44× 5.09×
> > q20 1.22× 1.56× 37.43× 68.20×
> > q21† 1.23× 1.31× 1.26× 1.27×
> > q22 1.33× 1.25× 1.30× 1.22×
> > q23 1.69× 3.32× 3.15× 12.10×
> > geomean 1.31× 1.44× 2.75× 3.41×
> >
> > † The benchmark's existing opt-in expression variants are used for these
> > headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
> > q21 use native datetime or regex/case behavior that can differ from Flink
> > at documented edge cases.
> >
> > [1] https://github.com/datafusion-contrib/StreamFusion
> > [2] https://streamfusion.tech/ .
> >
>

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Ran Tao <ch...@gmail.com>.
Hi Jacky,

Thanks for sharing StreamFusion! This looks like an interesting project.

I'm curious about how StreamFusion integrates with Flink's existing planner
and runtime, especially regarding stateful streaming, checkpointing, and
fault tolerance.

Also, has StreamFusion been used in production yet? It would be great to
hear about any real use cases and experiences.

Best Regards,
Ran Tao
https://github.com/taoran92


Jacky Lau <li...@gmail.com> 于2026年10月15日周四 11:14写道:

> Hi guys:
>      I want to introduce all of you a flink naive engine StreamFusion which
> is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It has
> good performance and supports lots of sql operator.
>     Will it be move it to flink subproject like
> flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
> Query Memory, off Memory, on Disk, off Disk, on
> q0 1.69× 1.39× 1.36× 1.40×
> q1† 1.40× 1.40× 1.38× 1.41×
> q2 1.26× 1.08× 1.11× 1.08×
> q3 1.03× 1.16× 1.08× 1.14×
> q4 1.83× 1.60× 8.92× 15.11×
> q5 1.41× 1.23× 4.38× 4.38×
> q7 1.30× 1.47× 7.41× 11.93×
> q8 1.27× 1.12× 2.33× 2.43×
> q9 1.22× 1.80× 17.83× 67.95×
> q10† 1.45× 1.94× 1.42× 1.40×
> q11 1.47× 1.50× 9.84× 10.03×
> q12 1.09× 1.13× 1.45× 1.33×
> q13 1.20× 1.14× 1.15× 1.22×
> q14† 1.47× 1.25× 1.49× 1.53×
> q15† 1.47× 1.22× 1.61× 1.32×
> q16† 1.17× 1.41× 1.49× 1.28×
> q17† 1.20× 1.18× 1.92× 1.65×
> q18 1.07× 1.37× 6.01× 8.92×
> q19 0.99× 2.82× 2.44× 5.09×
> q20 1.22× 1.56× 37.43× 68.20×
> q21† 1.23× 1.31× 1.26× 1.27×
> q22 1.33× 1.25× 1.30× 1.22×
> q23 1.69× 3.32× 3.15× 12.10×
> geomean 1.31× 1.44× 2.75× 3.41×
>
> † The benchmark's existing opt-in expression variants are used for these
> headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
> q21 use native datetime or regex/case behavior that can differ from Flink
> at documented edge cases.
>
> [1] https://github.com/datafusion-contrib/StreamFusion
> [2] https://streamfusion.tech/ .
>

Re: [DISCUSS] Flink Native Engine StreamFusion Based On DataFusion As Flink SubProject

Posted by Zhang Chaoming <zh...@outlook.com>.
Nice project!

> 2026年10月15日 11:13,Jacky Lau <li...@gmail.com> 写道:
> 
> Hi guys:
>     I want to introduce all of you a flink naive engine StreamFusion which
> is based on DataFusion, refers to Risingwave, Arroyo, DataFusion.It has
> good performance and supports lots of sql operator.
>    Will it be move it to flink subproject like
> flink-cdc/flink-kubernetes-operator. Anyone has a good advice?
> Query Memory, off Memory, on Disk, off Disk, on
> q0 1.69× 1.39× 1.36× 1.40×
> q1† 1.40× 1.40× 1.38× 1.41×
> q2 1.26× 1.08× 1.11× 1.08×
> q3 1.03× 1.16× 1.08× 1.14×
> q4 1.83× 1.60× 8.92× 15.11×
> q5 1.41× 1.23× 4.38× 4.38×
> q7 1.30× 1.47× 7.41× 11.93×
> q8 1.27× 1.12× 2.33× 2.43×
> q9 1.22× 1.80× 17.83× 67.95×
> q10† 1.45× 1.94× 1.42× 1.40×
> q11 1.47× 1.50× 9.84× 10.03×
> q12 1.09× 1.13× 1.45× 1.33×
> q13 1.20× 1.14× 1.15× 1.22×
> q14† 1.47× 1.25× 1.49× 1.53×
> q15† 1.47× 1.22× 1.61× 1.32×
> q16† 1.17× 1.41× 1.49× 1.28×
> q17† 1.20× 1.18× 1.92× 1.65×
> q18 1.07× 1.37× 6.01× 8.92×
> q19 0.99× 2.82× 2.44× 5.09×
> q20 1.22× 1.56× 37.43× 68.20×
> q21† 1.23× 1.31× 1.26× 1.27×
> q22 1.33× 1.25× 1.30× 1.22×
> q23 1.69× 3.32× 3.15× 12.10×
> geomean 1.31× 1.44× 2.75× 3.41×
> 
> † The benchmark's existing opt-in expression variants are used for these
> headline cells: q1 uses approximate decimal arithmetic; q10, q14–q17, and
> q21 use native datetime or regex/case behavior that can differ from Flink
> at documented edge cases.
> 
> [1] https://github.com/datafusion-contrib/StreamFusion
> [2] https://streamfusion.tech/ .