You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by WeitingChen <we...@apache.org> on 2024/04/09 06:18:53 UTC
Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Hi all,
We are excited to introduce a new Apache incubating project called Gluten.
Gluten serves as a middleware layer designed to offload Spark to native
engines like Velox or ClickHouse.
For more detailed information, please visit the project repository at
https://github.com/apache/incubator-gluten
Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
We eagerly await feedback from the Spark community.
Thanks,
Weiting.
Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Posted by Holden Karau <ho...@gmail.com>.
On Wed, Apr 10, 2024 at 9:54 PM Binwei Yang <fe...@apache.org> wrote:
>
> Gluten currently already support Velox backend and Clickhouse backend.
> data fusion support is also proposed but no one worked on it.
>
> Gluten isn't a POC. It's under actively developing but some companies
> already used it.
>
>
> On 2024/04/11 03:32:01 Dongjoon Hyun wrote:
> > I'm interested in your claim.
> >
> > Could you elaborate or provide some evidence for your claim, *a door for
> > all native libraries*, Binwei?
> >
> > For example, is there any POC for that claim? Maybe, did I miss something
> > in that SPIP?
>
I think the concern here is there are multiple different layers to get from
Spark -> Native code and ideally any changes we introduce in Spark would be
for common functionality that is useful across them (e.g. data fusion comet
& gluten & photon*, etc.)
* Photon being harder to guess at since it's closed source.
> >
> > Dongjoon.
> >
> > On Wed, Apr 10, 2024 at 8:19 PM Binwei Yang <fe...@apache.org> wrote:
> >
> > >
> > > The SPIP is not for current Gluten, but open a door for all native
> > > libraries and accelerators support.
> > >
> > > On 2024/04/11 00:27:43 Weiting Chen wrote:
> > > > Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> > > > For Spark version support, currently Gluten v1.1.1 support Spark3.2
> and
> > > 3.3.
> > > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> > > > Spark4.0 support for Gluten is depending on the release schedule in
> > > Spark community.
> > > >
> > > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > > > > Thank you for sharing, Weiting.
> > > > >
> > > > > Do you think you can share the future milestone of Apache Gluten?
> > > > > I'm wondering when the first stable release will come and how we
> can
> > > > > coordinate across the ASF communities.
> > > > >
> > > > > > This project is still under active development now, and doesn't
> have
> > > a
> > > > > stable release.
> > > > > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > > > >
> > > > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end
> of
> > > > > support.
> > > > > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release)
> is
> > > > > scheduled in October.
> > > > >
> > > > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only
> if
> > > there
> > > > > is something we need to do from Spark side.
> > > > >
> > > > > Thanks,
> > > > > Dongjoon.
> > > > >
> > > > >
> > > > > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <
> weitingchen@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are excited to introduce a new Apache incubating project
> called
> > > Gluten.
> > > > > > Gluten serves as a middleware layer designed to offload Spark to
> > > native
> > > > > > engines like Velox or ClickHouse.
> > > > > > For more detailed information, please visit the project
> repository at
> > > > > > https://github.com/apache/incubator-gluten
> > > > > >
> > > > > > Additionally, a new Spark SPIP related to Spark + Gluten
> > > collaboration has
> > > > > > been proposed at
> https://issues.apache.org/jira/browse/SPARK-47773.
> > > > > > We eagerly await feedback from the Spark community.
> > > > > >
> > > > > > Thanks,
> > > > > > Weiting.
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Posted by Binwei Yang <fe...@apache.org>.
Gluten currently already support Velox backend and Clickhouse backend. data fusion support is also proposed but no one worked on it.
Gluten isn't a POC. It's under actively developing but some companies already used it.
On 2024/04/11 03:32:01 Dongjoon Hyun wrote:
> I'm interested in your claim.
>
> Could you elaborate or provide some evidence for your claim, *a door for
> all native libraries*, Binwei?
>
> For example, is there any POC for that claim? Maybe, did I miss something
> in that SPIP?
>
> Dongjoon.
>
> On Wed, Apr 10, 2024 at 8:19 PM Binwei Yang <fe...@apache.org> wrote:
>
> >
> > The SPIP is not for current Gluten, but open a door for all native
> > libraries and accelerators support.
> >
> > On 2024/04/11 00:27:43 Weiting Chen wrote:
> > > Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> > > For Spark version support, currently Gluten v1.1.1 support Spark3.2 and
> > 3.3.
> > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> > > Spark4.0 support for Gluten is depending on the release schedule in
> > Spark community.
> > >
> > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > > > Thank you for sharing, Weiting.
> > > >
> > > > Do you think you can share the future milestone of Apache Gluten?
> > > > I'm wondering when the first stable release will come and how we can
> > > > coordinate across the ASF communities.
> > > >
> > > > > This project is still under active development now, and doesn't have
> > a
> > > > stable release.
> > > > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > > >
> > > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> > > > support.
> > > > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> > > > scheduled in October.
> > > >
> > > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if
> > there
> > > > is something we need to do from Spark side.
> > > >
> > > > Thanks,
> > > > Dongjoon.
> > > >
> > > >
> > > > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <we...@apache.org>
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We are excited to introduce a new Apache incubating project called
> > Gluten.
> > > > > Gluten serves as a middleware layer designed to offload Spark to
> > native
> > > > > engines like Velox or ClickHouse.
> > > > > For more detailed information, please visit the project repository at
> > > > > https://github.com/apache/incubator-gluten
> > > > >
> > > > > Additionally, a new Spark SPIP related to Spark + Gluten
> > collaboration has
> > > > > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > > > > We eagerly await feedback from the Spark community.
> > > > >
> > > > > Thanks,
> > > > > Weiting.
> > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Posted by Dongjoon Hyun <do...@gmail.com>.
I'm interested in your claim.
Could you elaborate or provide some evidence for your claim, *a door for
all native libraries*, Binwei?
For example, is there any POC for that claim? Maybe, did I miss something
in that SPIP?
Dongjoon.
On Wed, Apr 10, 2024 at 8:19 PM Binwei Yang <fe...@apache.org> wrote:
>
> The SPIP is not for current Gluten, but open a door for all native
> libraries and accelerators support.
>
> On 2024/04/11 00:27:43 Weiting Chen wrote:
> > Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> > For Spark version support, currently Gluten v1.1.1 support Spark3.2 and
> 3.3.
> > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> > Spark4.0 support for Gluten is depending on the release schedule in
> Spark community.
> >
> > On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > > Thank you for sharing, Weiting.
> > >
> > > Do you think you can share the future milestone of Apache Gluten?
> > > I'm wondering when the first stable release will come and how we can
> > > coordinate across the ASF communities.
> > >
> > > > This project is still under active development now, and doesn't have
> a
> > > stable release.
> > > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> > >
> > > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> > > support.
> > > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> > > scheduled in October.
> > >
> > > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if
> there
> > > is something we need to do from Spark side.
> > >
> > > Thanks,
> > > Dongjoon.
> > >
> > >
> > > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <we...@apache.org>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We are excited to introduce a new Apache incubating project called
> Gluten.
> > > > Gluten serves as a middleware layer designed to offload Spark to
> native
> > > > engines like Velox or ClickHouse.
> > > > For more detailed information, please visit the project repository at
> > > > https://github.com/apache/incubator-gluten
> > > >
> > > > Additionally, a new Spark SPIP related to Spark + Gluten
> collaboration has
> > > > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > > > We eagerly await feedback from the Spark community.
> > > >
> > > > Thanks,
> > > > Weiting.
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Posted by Binwei Yang <fe...@apache.org>.
The SPIP is not for current Gluten, but open a door for all native libraries and accelerators support.
On 2024/04/11 00:27:43 Weiting Chen wrote:
> Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
> For Spark version support, currently Gluten v1.1.1 support Spark3.2 and 3.3.
> We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
> Spark4.0 support for Gluten is depending on the release schedule in Spark community.
>
> On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> > Thank you for sharing, Weiting.
> >
> > Do you think you can share the future milestone of Apache Gluten?
> > I'm wondering when the first stable release will come and how we can
> > coordinate across the ASF communities.
> >
> > > This project is still under active development now, and doesn't have a
> > stable release.
> > > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
> >
> > In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> > support.
> > And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> > scheduled in October.
> >
> > For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there
> > is something we need to do from Spark side.
> >
> > Thanks,
> > Dongjoon.
> >
> >
> > On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <we...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > We are excited to introduce a new Apache incubating project called Gluten.
> > > Gluten serves as a middleware layer designed to offload Spark to native
> > > engines like Velox or ClickHouse.
> > > For more detailed information, please visit the project repository at
> > > https://github.com/apache/incubator-gluten
> > >
> > > Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
> > > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > > We eagerly await feedback from the Spark community.
> > >
> > > Thanks,
> > > Weiting.
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Posted by Weiting Chen <we...@apache.org>.
Yes, the 1st Apache release(v1.2.0) for Gluten will be in September.
For Spark version support, currently Gluten v1.1.1 support Spark3.2 and 3.3.
We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0.
Spark4.0 support for Gluten is depending on the release schedule in Spark community.
On 2024/04/09 07:14:13 Dongjoon Hyun wrote:
> Thank you for sharing, Weiting.
>
> Do you think you can share the future milestone of Apache Gluten?
> I'm wondering when the first stable release will come and how we can
> coordinate across the ASF communities.
>
> > This project is still under active development now, and doesn't have a
> stable release.
> > https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
>
> In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
> support.
> And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
> scheduled in October.
>
> For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there
> is something we need to do from Spark side.
>
> Thanks,
> Dongjoon.
>
>
> On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <we...@apache.org> wrote:
>
> > Hi all,
> >
> > We are excited to introduce a new Apache incubating project called Gluten.
> > Gluten serves as a middleware layer designed to offload Spark to native
> > engines like Velox or ClickHouse.
> > For more detailed information, please visit the project repository at
> > https://github.com/apache/incubator-gluten
> >
> > Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
> > been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> > We eagerly await feedback from the Spark community.
> >
> > Thanks,
> > Weiting.
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you for sharing, Weiting.
Do you think you can share the future milestone of Apache Gluten?
I'm wondering when the first stable release will come and how we can
coordinate across the ASF communities.
> This project is still under active development now, and doesn't have a
stable release.
> https://github.com/apache/incubator-gluten/releases/tag/v1.1.1
In the Apache Spark community, Apache Spark 3.2 and 3.3 is the end of
support.
And, 3.4 will have 3.4.3 next week and 3.4.4 (another EOL release) is
scheduled in October.
For the SPIP, I guess it's applicable for Apache Spark 4.0.0 only if there
is something we need to do from Spark side.
Thanks,
Dongjoon.
On Mon, Apr 8, 2024 at 11:19 PM WeitingChen <we...@apache.org> wrote:
> Hi all,
>
> We are excited to introduce a new Apache incubating project called Gluten.
> Gluten serves as a middleware layer designed to offload Spark to native
> engines like Velox or ClickHouse.
> For more detailed information, please visit the project repository at
> https://github.com/apache/incubator-gluten
>
> Additionally, a new Spark SPIP related to Spark + Gluten collaboration has
> been proposed at https://issues.apache.org/jira/browse/SPARK-47773.
> We eagerly await feedback from the Spark community.
>
> Thanks,
> Weiting.
>
>