You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Lari Hotari <lh...@apache.org> on 2021/12/22 07:31:17 UTC
[DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Dear Pulsar community members,
PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
repositories" was created in April 2020. The repositories for
pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
ago [2].
I'd like to propose that we continue with the PIP-62 plan asap, before
Apache Pulsar 2.10.0 is released.
I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
waiting for the Trino Pulsar PR [3] being completed.
I am volunteering to making the changes as per the PIP-62 plan.
When can we proceed with the plan? Do we need an explicit vote on the
mailing list about this?
BR,
Lari
[1]
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
[2]
https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
[3] https://github.com/trinodb/trino/pull/8020
Other email threads:
* [Discuss] Don't include presto/trino in the normal Pulsar distribution -
https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
* Updates on Presto connector for PIP-62 -
https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Lari Hotari <lh...@apache.org>.
Is there a plan to make progress with the Trino PR https://github.com/trinodb/trino/pull/8020 which would replace Pulsar SQL?
Pulsar SQL is not compatibile with Java 11.0.14.1 since Presto 332 fails to parse this type of version string. That issue is reported as https://github.com/apache/pulsar/issues/14951 .
The PR to fix that is https://github.com/apache/pulsar/pull/14953 . Presto 334 removes the Java version parser class completely and therefore it fixes the issue. Presto 334 requires Java 11 from what I've heard so merging this PR would drop support for Java 8 in Pulsar SQL.
I would assume that it's fine for doing this in the master branch.
Are there any ideas how to resolve https://github.com/apache/pulsar/issues/14951 in some other way?
BR, Lari
On 2021/12/23 08:42:57 Lari Hotari wrote:
> > I believe we want to keep SQL until the code change lands in Trino.
> Because
> there are still users using this component.
>
> There have been multiple public dev@pulsar.apache.org discussions about
> PIP-62 which was initialized in April 2020.
> Those users using Pulsar SQL can keep on using Pulsar 2.8.x or 2.9.x until
> the feature is available in Trino. Isn't that a viable option?
>
> The Pulsar community is here to help with the work. What is the status of
> Trino work? Are there tasks that need help? Is there any ETA for the
> landing of code in Trino?
> Could we resolve this together?
>
> BR, Lari
>
> On Thu, Dec 23, 2021 at 2:39 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > Sorry. When you say "we discussed", who are "we"? Is it DataStax?
> >
> > I believe we want to keep SQL until the code change lands in Trino. Because
> > there are still users using this component.
> >
> > - Sijie
> >
> > On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Lari,
> > >
> > > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <lhotari@apache.org
> > >
> > > ha scritto:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > > separate
> > > > repositories" was created in April 2020. The repositories for
> > > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> > > year
> > > > ago [2].
> > > >
> > > > I'd like to propose that we continue with the PIP-62 plan asap, before
> > > > Apache Pulsar 2.10.0 is released.
> > > >
> > > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > > > waiting for the Trino Pulsar PR [3] being completed.
> > > >
> > > > I am volunteering to making the changes as per the PIP-62 plan.
> > > > When can we proceed with the plan? Do we need an explicit vote on the
> > > > mailing list about this?
> > > >
> > >
> > > We discussed this many times in the past year.
> > > I believe that there is no need to wait
> > >
> > > Enrico
> > >
> > >
> > >
> > > >
> > > > BR,
> > > >
> > > > Lari
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > > [2]
> > > >
> > > >
> > >
> > https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > > [3] https://github.com/trinodb/trino/pull/8020
> > > >
> > > > Other email threads:
> > > > * [Discuss] Don't include presto/trino in the normal Pulsar
> > distribution
> > > -
> > > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > > * Updates on Presto connector for PIP-62 -
> > > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > > >
> > >
> >
>
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Michael Marshall <mm...@apache.org>.
I think I misunderstood the current state of PIP 62. Once we clarify
the plan, it'd be helpful to update the wiki.
Thanks,
Michael
On Thu, Dec 23, 2021 at 2:43 AM Lari Hotari <La...@hotari.net> wrote:
>
> > I believe we want to keep SQL until the code change lands in Trino.
> Because
> there are still users using this component.
>
> There have been multiple public dev@pulsar.apache.org discussions about
> PIP-62 which was initialized in April 2020.
> Those users using Pulsar SQL can keep on using Pulsar 2.8.x or 2.9.x until
> the feature is available in Trino. Isn't that a viable option?
>
> The Pulsar community is here to help with the work. What is the status of
> Trino work? Are there tasks that need help? Is there any ETA for the
> landing of code in Trino?
> Could we resolve this together?
>
> BR, Lari
>
> On Thu, Dec 23, 2021 at 2:39 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > Sorry. When you say "we discussed", who are "we"? Is it DataStax?
> >
> > I believe we want to keep SQL until the code change lands in Trino. Because
> > there are still users using this component.
> >
> > - Sijie
> >
> > On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Lari,
> > >
> > > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <lhotari@apache.org
> > >
> > > ha scritto:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > > separate
> > > > repositories" was created in April 2020. The repositories for
> > > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> > > year
> > > > ago [2].
> > > >
> > > > I'd like to propose that we continue with the PIP-62 plan asap, before
> > > > Apache Pulsar 2.10.0 is released.
> > > >
> > > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > > > waiting for the Trino Pulsar PR [3] being completed.
> > > >
> > > > I am volunteering to making the changes as per the PIP-62 plan.
> > > > When can we proceed with the plan? Do we need an explicit vote on the
> > > > mailing list about this?
> > > >
> > >
> > > We discussed this many times in the past year.
> > > I believe that there is no need to wait
> > >
> > > Enrico
> > >
> > >
> > >
> > > >
> > > > BR,
> > > >
> > > > Lari
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > > [2]
> > > >
> > > >
> > >
> > https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > > [3] https://github.com/trinodb/trino/pull/8020
> > > >
> > > > Other email threads:
> > > > * [Discuss] Don't include presto/trino in the normal Pulsar
> > distribution
> > > -
> > > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > > * Updates on Presto connector for PIP-62 -
> > > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > > >
> > >
> >
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Lari Hotari <La...@hotari.net>.
> I believe we want to keep SQL until the code change lands in Trino.
Because
there are still users using this component.
There have been multiple public dev@pulsar.apache.org discussions about
PIP-62 which was initialized in April 2020.
Those users using Pulsar SQL can keep on using Pulsar 2.8.x or 2.9.x until
the feature is available in Trino. Isn't that a viable option?
The Pulsar community is here to help with the work. What is the status of
Trino work? Are there tasks that need help? Is there any ETA for the
landing of code in Trino?
Could we resolve this together?
BR, Lari
On Thu, Dec 23, 2021 at 2:39 AM Sijie Guo <gu...@gmail.com> wrote:
> Sorry. When you say "we discussed", who are "we"? Is it DataStax?
>
> I believe we want to keep SQL until the code change lands in Trino. Because
> there are still users using this component.
>
> - Sijie
>
> On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Lari,
> >
> > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <lhotari@apache.org
> >
> > ha scritto:
> >
> > > Dear Pulsar community members,
> > >
> > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > separate
> > > repositories" was created in April 2020. The repositories for
> > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> > year
> > > ago [2].
> > >
> > > I'd like to propose that we continue with the PIP-62 plan asap, before
> > > Apache Pulsar 2.10.0 is released.
> > >
> > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > > waiting for the Trino Pulsar PR [3] being completed.
> > >
> > > I am volunteering to making the changes as per the PIP-62 plan.
> > > When can we proceed with the plan? Do we need an explicit vote on the
> > > mailing list about this?
> > >
> >
> > We discussed this many times in the past year.
> > I believe that there is no need to wait
> >
> > Enrico
> >
> >
> >
> > >
> > > BR,
> > >
> > > Lari
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > [2]
> > >
> > >
> >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > [3] https://github.com/trinodb/trino/pull/8020
> > >
> > > Other email threads:
> > > * [Discuss] Don't include presto/trino in the normal Pulsar
> distribution
> > -
> > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > * Updates on Presto connector for PIP-62 -
> > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > >
> >
>
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Enrico Olivelli <eo...@gmail.com>.
Sijie,
> Sorry. When you say "we discussed", who are "we"? Is it DataStax?
There have been discussions on the ML and we discussed this also in
Community meetings.
So "we" is this community
> I believe we want to keep SQL until the code change lands in Trino.
Because
> there are still users using this component.
If we don't want to remove Presto then we have to at least move to the
latest version, that requires JDK11.
My understanding is that Lari is willing to help in completing this work.
When we discussed this, probably something like 1 year ago, Lari did not
have write permissions and he could not help.
There is not only Presto, we had to move all the Connectors and so change
the build process for the Pulsar "-all" docker image
We can defer Presto and move forward with the Connectors
Enrico
Il giorno gio 23 dic 2021 alle ore 06:34 Michael Marshall <
mmarshall@apache.org> ha scritto:
> > I am volunteering to making the changes as per the PIP-62 plan.
>
> I think Lari is proposing to complete the work to move the
> Pulsar SQL code base to the repositories defined in PIP 62.
>
> Assuming that is what he meant, I am +1 on completing that now.
> Based on reading through the mailing list threads Lari referenced, we
> delayed moving the code base to the other repos with the hope that
> contributing the code to Trino/Presto would be done before the 2.8.0
> release. I think we should move forward with the initial PIP instead
> of waiting for the Trino PR to get merged.
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:39 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > Sorry. When you say "we discussed", who are "we"? Is it DataStax?
> >
> > I believe we want to keep SQL until the code change lands in Trino.
> Because
> > there are still users using this component.
> >
> > - Sijie
> >
> > On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Lari,
> > >
> > > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <
> lhotari@apache.org>
> > > ha scritto:
> > >
> > > > Dear Pulsar community members,
> > > >
> > > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > > separate
> > > > repositories" was created in April 2020. The repositories for
> > > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about
> a
> > > year
> > > > ago [2].
> > > >
> > > > I'd like to propose that we continue with the PIP-62 plan asap,
> before
> > > > Apache Pulsar 2.10.0 is released.
> > > >
> > > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar
> without
> > > > waiting for the Trino Pulsar PR [3] being completed.
> > > >
> > > > I am volunteering to making the changes as per the PIP-62 plan.
> > > > When can we proceed with the plan? Do we need an explicit vote on the
> > > > mailing list about this?
> > > >
> > >
> > > We discussed this many times in the past year.
> > > I believe that there is no need to wait
> > >
> > > Enrico
> > >
> > >
> > >
> > > >
> > > > BR,
> > > >
> > > > Lari
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > > [2]
> > > >
> > > >
> > >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > > [3] https://github.com/trinodb/trino/pull/8020
> > > >
> > > > Other email threads:
> > > > * [Discuss] Don't include presto/trino in the normal Pulsar
> distribution
> > > -
> > > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > > * Updates on Presto connector for PIP-62 -
> > > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > > >
> > >
>
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Michael Marshall <mm...@apache.org>.
> I am volunteering to making the changes as per the PIP-62 plan.
I think Lari is proposing to complete the work to move the
Pulsar SQL code base to the repositories defined in PIP 62.
Assuming that is what he meant, I am +1 on completing that now.
Based on reading through the mailing list threads Lari referenced, we
delayed moving the code base to the other repos with the hope that
contributing the code to Trino/Presto would be done before the 2.8.0
release. I think we should move forward with the initial PIP instead
of waiting for the Trino PR to get merged.
- Michael
On Wed, Dec 22, 2021 at 6:39 PM Sijie Guo <gu...@gmail.com> wrote:
>
> Sorry. When you say "we discussed", who are "we"? Is it DataStax?
>
> I believe we want to keep SQL until the code change lands in Trino. Because
> there are still users using this component.
>
> - Sijie
>
> On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Lari,
> >
> > Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <lh...@apache.org>
> > ha scritto:
> >
> > > Dear Pulsar community members,
> > >
> > > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> > separate
> > > repositories" was created in April 2020. The repositories for
> > > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> > year
> > > ago [2].
> > >
> > > I'd like to propose that we continue with the PIP-62 plan asap, before
> > > Apache Pulsar 2.10.0 is released.
> > >
> > > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > > waiting for the Trino Pulsar PR [3] being completed.
> > >
> > > I am volunteering to making the changes as per the PIP-62 plan.
> > > When can we proceed with the plan? Do we need an explicit vote on the
> > > mailing list about this?
> > >
> >
> > We discussed this many times in the past year.
> > I believe that there is no need to wait
> >
> > Enrico
> >
> >
> >
> > >
> > > BR,
> > >
> > > Lari
> > >
> > > [1]
> > >
> > >
> > https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > [2]
> > >
> > >
> > https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > > [3] https://github.com/trinodb/trino/pull/8020
> > >
> > > Other email threads:
> > > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> > -
> > > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > > * Updates on Presto connector for PIP-62 -
> > > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> > >
> >
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Sijie Guo <gu...@gmail.com>.
Sorry. When you say "we discussed", who are "we"? Is it DataStax?
I believe we want to keep SQL until the code change lands in Trino. Because
there are still users using this component.
- Sijie
On Tue, Dec 21, 2021 at 11:34 PM Enrico Olivelli <eo...@gmail.com>
wrote:
> Lari,
>
> Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <lh...@apache.org>
> ha scritto:
>
> > Dear Pulsar community members,
> >
> > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> separate
> > repositories" was created in April 2020. The repositories for
> > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> year
> > ago [2].
> >
> > I'd like to propose that we continue with the PIP-62 plan asap, before
> > Apache Pulsar 2.10.0 is released.
> >
> > I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> > waiting for the Trino Pulsar PR [3] being completed.
> >
> > I am volunteering to making the changes as per the PIP-62 plan.
> > When can we proceed with the plan? Do we need an explicit vote on the
> > mailing list about this?
> >
>
> We discussed this many times in the past year.
> I believe that there is no need to wait
>
> Enrico
>
>
>
> >
> > BR,
> >
> > Lari
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> > [3] https://github.com/trinodb/trino/pull/8020
> >
> > Other email threads:
> > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> -
> > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > * Updates on Presto connector for PIP-62 -
> > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> >
>
Re: [DISCUSS] Proceeding with PIP-62 plan, before Apache Pulsar 2.10.0 is released
Posted by Enrico Olivelli <eo...@gmail.com>.
Lari,
Il giorno mer 22 dic 2021 alle ore 08:31 Lari Hotari <lh...@apache.org>
ha scritto:
> Dear Pulsar community members,
>
> PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
> repositories" was created in April 2020. The repositories for
> pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
> ago [2].
>
> I'd like to propose that we continue with the PIP-62 plan asap, before
> Apache Pulsar 2.10.0 is released.
>
> I'm also suggesting that we drop Pulsar SQL from apache/pulsar without
> waiting for the Trino Pulsar PR [3] being completed.
>
> I am volunteering to making the changes as per the PIP-62 plan.
> When can we proceed with the plan? Do we need an explicit vote on the
> mailing list about this?
>
We discussed this many times in the past year.
I believe that there is no need to wait
Enrico
>
> BR,
>
> Lari
>
> [1]
>
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> [2]
>
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> [3] https://github.com/trinodb/trino/pull/8020
>
> Other email threads:
> * [Discuss] Don't include presto/trino in the normal Pulsar distribution -
> https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> * Updates on Presto connector for PIP-62 -
> https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
>