You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pulsar.apache.org by Nicolò Boschi <bo...@gmail.com> on 2021/11/02 13:59:09 UTC

Re: Dropping Presto SQL in 2.9.0 - status ?

Resurrecting this thread.

2.9 is almost released and it hasn't been merged yet

Extending the discussion to other connectors, it looks like there has been
no progress on PIP-62.
My concern is that a lot of Pulsar IO connectors dependencies we are
running are obsolete with several security reports

I see there are interesting comments in the issue (
https://github.com/apache/pulsar/issues/10219) and Sijie exported the
pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
outdated

From my point of view, we have to:
- reimport all the connectors source codes with newest ones (including
integration tests)
- add periodic CI jobs for connectors to run against master, 2.9-latest,
2.8-latest, 2.7-latest to verify breaking changes
- define a release cycle/management for connectors (we should improve the
PIP doc). IMO it's not clear if each connector will have its own release
versions and how we'll handle it (git tags, artifacts deployment..)
- update pulsar release script in order to get the connectors artifacts
(retrieving the .nar or building it from source?)
- update docs
- remove pulsar-io dir from Pulsar repo

It's the perfect timing to schedule this work for 2.10

What is missing? How's the situation? Is there a roadblock I haven't seen?
I think it's better to take another discussion for Presto since it will
come to another end


Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
eolivelli@gmail.com> ha scritto:

> Sijie
>
> Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
>
> > You can follow the progress at
> https://github.com/trinodb/trino/pull/8020.
> >
>
> Thanks for the pointer
>
> >
> > The original code doesn't conform to TrinoDB's standard. Marvin is
> > actively following up on that.
> >
> > Our goal is still to get this completed as part of the 2.9 release.
> >
>
> Wonderful
>
> Thanks
> Enrico
>
> >
> > - Sijie
> >
> > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > >
> > > Hello,
> > > How is the Presto work going ?
> > > IIRC the plan was to remove it from the Pulsar code base and let it be
> > > hosted at Trino.
> > >
> > > If this is not going to happen within the 2.9.0 release timeline
> > > (September?) I would prefer to upgrade to "Trino".
> > > Probably we will have a downside problem that recent versions of
> > > Presto/Trino do not work on JDK8 but only on JDK11.
> > >
> > > I believe that in that case we could open a separate thread to say that
> > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and it is
> the
> > > preferred version, as it is the version used in the Docker images), so
> > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > >
> > > My primary concern is that the version of Presto that we are running is
> > > obsolete and there are several security reports against it or its third
> > > party dependencies.
> > >
> > > Thoughts ?
> > >
> > > Enrico
> >
>


-- 
Nicolò Boschi

Re: Dropping Presto SQL in 2.9.0 - status ?

Posted by Neng Lu <nl...@apache.org>.

Just curious to learn is there any progress on moving all the connectors into separate repos?
Maybe I can help if the decision is finalized.

On 2021/11/17 06:18:52 Lari Hotari wrote:
> Dear Pulsar community members,
> 
> PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
> repositories" was created in April 2020. The repositories for
> pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
> ago [2].
> 
> What is the current roadmap for completing PIP-62 and moving
> pulsar-connectors and pulsar-sql out of apache/pulsar repository?
> 
> BR,
> 
> Lari
> 
> [1]
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> [2]
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> 
> Other email threads:
> * [Discuss] Don't include presto/trino in the normal Pulsar distribution -
> https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> * Updates on Presto connector for PIP-62 -
> https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> 
> On Tue, Nov 2, 2021 at 3:59 PM Nicolò Boschi <bo...@gmail.com> wrote:
> 
> > Resurrecting this thread.
> >
> > 2.9 is almost released and it hasn't been merged yet
> >
> > Extending the discussion to other connectors, it looks like there has been
> > no progress on PIP-62.
> > My concern is that a lot of Pulsar IO connectors dependencies we are
> > running are obsolete with several security reports
> >
> > I see there are interesting comments in the issue (
> > https://github.com/apache/pulsar/issues/10219) and Sijie exported the
> > pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
> > outdated
> >
> > From my point of view, we have to:
> > - reimport all the connectors source codes with newest ones (including
> > integration tests)
> > - add periodic CI jobs for connectors to run against master, 2.9-latest,
> > 2.8-latest, 2.7-latest to verify breaking changes
> > - define a release cycle/management for connectors (we should improve the
> > PIP doc). IMO it's not clear if each connector will have its own release
> > versions and how we'll handle it (git tags, artifacts deployment..)
> > - update pulsar release script in order to get the connectors artifacts
> > (retrieving the .nar or building it from source?)
> > - update docs
> > - remove pulsar-io dir from Pulsar repo
> >
> > It's the perfect timing to schedule this work for 2.10
> >
> > What is missing? How's the situation? Is there a roadblock I haven't seen?
> > I think it's better to take another discussion for Presto since it will
> > come to another end
> >
> >
> > Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
> > eolivelli@gmail.com> ha scritto:
> >
> > > Sijie
> > >
> > > Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
> > >
> > > > You can follow the progress at
> > > https://github.com/trinodb/trino/pull/8020.
> > > >
> > >
> > > Thanks for the pointer
> > >
> > > >
> > > > The original code doesn't conform to TrinoDB's standard. Marvin is
> > > > actively following up on that.
> > > >
> > > > Our goal is still to get this completed as part of the 2.9 release.
> > > >
> > >
> > > Wonderful
> > >
> > > Thanks
> > > Enrico
> > >
> > > >
> > > > - Sijie
> > > >
> > > > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <eo...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hello,
> > > > > How is the Presto work going ?
> > > > > IIRC the plan was to remove it from the Pulsar code base and let it
> > be
> > > > > hosted at Trino.
> > > > >
> > > > > If this is not going to happen within the 2.9.0 release timeline
> > > > > (September?) I would prefer to upgrade to "Trino".
> > > > > Probably we will have a downside problem that recent versions of
> > > > > Presto/Trino do not work on JDK8 but only on JDK11.
> > > > >
> > > > > I believe that in that case we could open a separate thread to say
> > that
> > > > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and it is
> > > the
> > > > > preferred version, as it is the version used in the Docker images),
> > so
> > > > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > > > >
> > > > > My primary concern is that the version of Presto that we are running
> > is
> > > > > obsolete and there are several security reports against it or its
> > third
> > > > > party dependencies.
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > Enrico
> > > >
> > >
> >
> >
> > --
> > Nicolò Boschi
> >
>

Re: Dropping Presto SQL in 2.9.0 - status ?

Posted by Lari Hotari <La...@hotari.net>.

Thanks for the update Marvin,
Great work on the Trino PR! It's been a lot of work to get it to match the
Trino code conventions.
Is there a lot of work remaining?

-Lari

On Wed, Nov 17, 2021 at 3:40 PM Zhengxin Cai <ca...@gmail.com> wrote:

> Hi there,
> I think the pr is still open, https://github.com/trinodb/trino/pull/8020,
> will try to push it.
> But even after the pr is merged, I actually still think we might still want
> to keep a copy of the connector in Pulsar repo and push changes to Trino
> repo periodically, as this will allow much faster bug fix and feature
> iteration.
> Best,
> Marvin,
>
> Lari Hotari <lh...@apache.org> 于2021年11月17日周三 下午2:19写道：
>
> > Dear Pulsar community members,
> >
> > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> separate
> > repositories" was created in April 2020. The repositories for
> > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> year
> > ago [2].
> >
> > What is the current roadmap for completing PIP-62 and moving
> > pulsar-connectors and pulsar-sql out of apache/pulsar repository?
> >
> > BR,
> >
> > Lari
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> >
> > Other email threads:
> > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> -
> > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > * Updates on Presto connector for PIP-62 -
> > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> >
> > On Tue, Nov 2, 2021 at 3:59 PM Nicolò Boschi <bo...@gmail.com>
> wrote:
> >
> > > Resurrecting this thread.
> > >
> > > 2.9 is almost released and it hasn't been merged yet
> > >
> > > Extending the discussion to other connectors, it looks like there has
> > been
> > > no progress on PIP-62.
> > > My concern is that a lot of Pulsar IO connectors dependencies we are
> > > running are obsolete with several security reports
> > >
> > > I see there are interesting comments in the issue (
> > > https://github.com/apache/pulsar/issues/10219) and Sijie exported the
> > > pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
> > > outdated
> > >
> > > From my point of view, we have to:
> > > - reimport all the connectors source codes with newest ones (including
> > > integration tests)
> > > - add periodic CI jobs for connectors to run against master,
> 2.9-latest,
> > > 2.8-latest, 2.7-latest to verify breaking changes
> > > - define a release cycle/management for connectors (we should improve
> the
> > > PIP doc). IMO it's not clear if each connector will have its own
> release
> > > versions and how we'll handle it (git tags, artifacts deployment..)
> > > - update pulsar release script in order to get the connectors artifacts
> > > (retrieving the .nar or building it from source?)
> > > - update docs
> > > - remove pulsar-io dir from Pulsar repo
> > >
> > > It's the perfect timing to schedule this work for 2.10
> > >
> > > What is missing? How's the situation? Is there a roadblock I haven't
> > seen?
> > > I think it's better to take another discussion for Presto since it will
> > > come to another end
> > >
> > >
> > > Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
> > > eolivelli@gmail.com> ha scritto:
> > >
> > > > Sijie
> > > >
> > > > Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
> > > >
> > > > > You can follow the progress at
> > > > https://github.com/trinodb/trino/pull/8020.
> > > > >
> > > >
> > > > Thanks for the pointer
> > > >
> > > > >
> > > > > The original code doesn't conform to TrinoDB's standard. Marvin is
> > > > > actively following up on that.
> > > > >
> > > > > Our goal is still to get this completed as part of the 2.9 release.
> > > > >
> > > >
> > > > Wonderful
> > > >
> > > > Thanks
> > > > Enrico
> > > >
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <
> eolivelli@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hello,
> > > > > > How is the Presto work going ?
> > > > > > IIRC the plan was to remove it from the Pulsar code base and let
> it
> > > be
> > > > > > hosted at Trino.
> > > > > >
> > > > > > If this is not going to happen within the 2.9.0 release timeline
> > > > > > (September?) I would prefer to upgrade to "Trino".
> > > > > > Probably we will have a downside problem that recent versions of
> > > > > > Presto/Trino do not work on JDK8 but only on JDK11.
> > > > > >
> > > > > > I believe that in that case we could open a separate thread to
> say
> > > that
> > > > > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > > > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and
> it
> > is
> > > > the
> > > > > > preferred version, as it is the version used in the Docker
> images),
> > > so
> > > > > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > > > > >
> > > > > > My primary concern is that the version of Presto that we are
> > running
> > > is
> > > > > > obsolete and there are several security reports against it or its
> > > third
> > > > > > party dependencies.
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > Enrico
> > > > >
> > > >
> > >
> > >
> > > --
> > > Nicolò Boschi
> > >
> >
>

Re: Dropping Presto SQL in 2.9.0 - status ?

Posted by Enrico Olivelli <eo...@gmail.com>.

Marvin,

Il giorno mer 17 nov 2021 alle ore 14:39 Zhengxin Cai <ca...@gmail.com>
ha scritto:

> Hi there,
> I think the pr is still open, https://github.com/trinodb/trino/pull/8020,
> will try to push it.
> But even after the pr is merged, I actually still think we might still want
> to keep a copy of the connector in Pulsar repo and push changes to Trino
> repo periodically, as this will allow much faster bug fix and feature
> iteration.
>

That's not the plan.
We want to get rid of Presto/Trino from the Pulsar repo.

if you want to keep it then we must upgrade to Trino.

Currently we have a very outdated version of Presto, that brings in a few
obsolete dependencies tagged with CVEs.

Please note that more recent versions of Presto/Trino require JDK11,
so a switch to Trino may be feasible for Pulsar 2.10 if we drop support for
JDK8

Enrico



> Best,
> Marvin,
>
> Lari Hotari <lh...@apache.org> 于2021年11月17日周三 下午2:19写道：
>
> > Dear Pulsar community members,
> >
> > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> separate
> > repositories" was created in April 2020. The repositories for
> > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> year
> > ago [2].
> >
> > What is the current roadmap for completing PIP-62 and moving
> > pulsar-connectors and pulsar-sql out of apache/pulsar repository?
> >
> > BR,
> >
> > Lari
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> >
> > Other email threads:
> > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> -
> > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > * Updates on Presto connector for PIP-62 -
> > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> >
> > On Tue, Nov 2, 2021 at 3:59 PM Nicolò Boschi <bo...@gmail.com>
> wrote:
> >
> > > Resurrecting this thread.
> > >
> > > 2.9 is almost released and it hasn't been merged yet
> > >
> > > Extending the discussion to other connectors, it looks like there has
> > been
> > > no progress on PIP-62.
> > > My concern is that a lot of Pulsar IO connectors dependencies we are
> > > running are obsolete with several security reports
> > >
> > > I see there are interesting comments in the issue (
> > > https://github.com/apache/pulsar/issues/10219) and Sijie exported the
> > > pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
> > > outdated
> > >
> > > From my point of view, we have to:
> > > - reimport all the connectors source codes with newest ones (including
> > > integration tests)
> > > - add periodic CI jobs for connectors to run against master,
> 2.9-latest,
> > > 2.8-latest, 2.7-latest to verify breaking changes
> > > - define a release cycle/management for connectors (we should improve
> the
> > > PIP doc). IMO it's not clear if each connector will have its own
> release
> > > versions and how we'll handle it (git tags, artifacts deployment..)
> > > - update pulsar release script in order to get the connectors artifacts
> > > (retrieving the .nar or building it from source?)
> > > - update docs
> > > - remove pulsar-io dir from Pulsar repo
> > >
> > > It's the perfect timing to schedule this work for 2.10
> > >
> > > What is missing? How's the situation? Is there a roadblock I haven't
> > seen?
> > > I think it's better to take another discussion for Presto since it will
> > > come to another end
> > >
> > >
> > > Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
> > > eolivelli@gmail.com> ha scritto:
> > >
> > > > Sijie
> > > >
> > > > Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
> > > >
> > > > > You can follow the progress at
> > > > https://github.com/trinodb/trino/pull/8020.
> > > > >
> > > >
> > > > Thanks for the pointer
> > > >
> > > > >
> > > > > The original code doesn't conform to TrinoDB's standard. Marvin is
> > > > > actively following up on that.
> > > > >
> > > > > Our goal is still to get this completed as part of the 2.9 release.
> > > > >
> > > >
> > > > Wonderful
> > > >
> > > > Thanks
> > > > Enrico
> > > >
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <
> eolivelli@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hello,
> > > > > > How is the Presto work going ?
> > > > > > IIRC the plan was to remove it from the Pulsar code base and let
> it
> > > be
> > > > > > hosted at Trino.
> > > > > >
> > > > > > If this is not going to happen within the 2.9.0 release timeline
> > > > > > (September?) I would prefer to upgrade to "Trino".
> > > > > > Probably we will have a downside problem that recent versions of
> > > > > > Presto/Trino do not work on JDK8 but only on JDK11.
> > > > > >
> > > > > > I believe that in that case we could open a separate thread to
> say
> > > that
> > > > > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > > > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and
> it
> > is
> > > > the
> > > > > > preferred version, as it is the version used in the Docker
> images),
> > > so
> > > > > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > > > > >
> > > > > > My primary concern is that the version of Presto that we are
> > running
> > > is
> > > > > > obsolete and there are several security reports against it or its
> > > third
> > > > > > party dependencies.
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > Enrico
> > > > >
> > > >
> > >
> > >
> > > --
> > > Nicolò Boschi
> > >
> >
>

Re: Dropping Presto SQL in 2.9.0 - status ?

Posted by Lari Hotari <La...@hotari.net>.

Hi Marvin,

Great work on the Trino PR! It's been a lot of work to get it to match the
Trino code conventions.

I hope we could drop Presto & Pulsar SQL from the apache/pulsar code
repository as planned in PIP-62[1], "PIP 62: Move connectors, adapters and
Pulsar Presto to separate repositories", which was created in April
2020. Let's work together to complete this effort.

Is there anything that others could help with to complete the Trino PR
https://github.com/trinodb/trino/pull/8020 ?

BR,

Lari

[1]
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-
Presto-to-separate-repositories


On Wed, Nov 17, 2021 at 3:40 PM Zhengxin Cai <ca...@gmail.com> wrote:

> Hi there,
> I think the pr is still open, https://github.com/trinodb/trino/pull/8020,
> will try to push it.
> But even after the pr is merged, I actually still think we might still want
> to keep a copy of the connector in Pulsar repo and push changes to Trino
> repo periodically, as this will allow much faster bug fix and feature
> iteration.
> Best,
> Marvin,
>
> Lari Hotari <lh...@apache.org> 于2021年11月17日周三 下午2:19写道：
>
> > Dear Pulsar community members,
> >
> > PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to
> separate
> > repositories" was created in April 2020. The repositories for
> > pulsar-connectors, pulsar-adapters and pulsar-sql were created about a
> year
> > ago [2].
> >
> > What is the current roadmap for completing PIP-62 and moving
> > pulsar-connectors and pulsar-sql out of apache/pulsar repository?
> >
> > BR,
> >
> > Lari
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
> >
> > Other email threads:
> > * [Discuss] Don't include presto/trino in the normal Pulsar distribution
> -
> > https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> > * Updates on Presto connector for PIP-62 -
> > https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
> >
> > On Tue, Nov 2, 2021 at 3:59 PM Nicolò Boschi <bo...@gmail.com>
> wrote:
> >
> > > Resurrecting this thread.
> > >
> > > 2.9 is almost released and it hasn't been merged yet
> > >
> > > Extending the discussion to other connectors, it looks like there has
> > been
> > > no progress on PIP-62.
> > > My concern is that a lot of Pulsar IO connectors dependencies we are
> > > running are obsolete with several security reports
> > >
> > > I see there are interesting comments in the issue (
> > > https://github.com/apache/pulsar/issues/10219) and Sijie exported the
> > > pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
> > > outdated
> > >
> > > From my point of view, we have to:
> > > - reimport all the connectors source codes with newest ones (including
> > > integration tests)
> > > - add periodic CI jobs for connectors to run against master,
> 2.9-latest,
> > > 2.8-latest, 2.7-latest to verify breaking changes
> > > - define a release cycle/management for connectors (we should improve
> the
> > > PIP doc). IMO it's not clear if each connector will have its own
> release
> > > versions and how we'll handle it (git tags, artifacts deployment..)
> > > - update pulsar release script in order to get the connectors artifacts
> > > (retrieving the .nar or building it from source?)
> > > - update docs
> > > - remove pulsar-io dir from Pulsar repo
> > >
> > > It's the perfect timing to schedule this work for 2.10
> > >
> > > What is missing? How's the situation? Is there a roadblock I haven't
> > seen?
> > > I think it's better to take another discussion for Presto since it will
> > > come to another end
> > >
> > >
> > > Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
> > > eolivelli@gmail.com> ha scritto:
> > >
> > > > Sijie
> > > >
> > > > Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
> > > >
> > > > > You can follow the progress at
> > > > https://github.com/trinodb/trino/pull/8020.
> > > > >
> > > >
> > > > Thanks for the pointer
> > > >
> > > > >
> > > > > The original code doesn't conform to TrinoDB's standard. Marvin is
> > > > > actively following up on that.
> > > > >
> > > > > Our goal is still to get this completed as part of the 2.9 release.
> > > > >
> > > >
> > > > Wonderful
> > > >
> > > > Thanks
> > > > Enrico
> > > >
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <
> eolivelli@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hello,
> > > > > > How is the Presto work going ?
> > > > > > IIRC the plan was to remove it from the Pulsar code base and let
> it
> > > be
> > > > > > hosted at Trino.
> > > > > >
> > > > > > If this is not going to happen within the 2.9.0 release timeline
> > > > > > (September?) I would prefer to upgrade to "Trino".
> > > > > > Probably we will have a downside problem that recent versions of
> > > > > > Presto/Trino do not work on JDK8 but only on JDK11.
> > > > > >
> > > > > > I believe that in that case we could open a separate thread to
> say
> > > that
> > > > > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > > > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and
> it
> > is
> > > > the
> > > > > > preferred version, as it is the version used in the Docker
> images),
> > > so
> > > > > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > > > > >
> > > > > > My primary concern is that the version of Presto that we are
> > running
> > > is
> > > > > > obsolete and there are several security reports against it or its
> > > third
> > > > > > party dependencies.
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > Enrico
> > > > >
> > > >
> > >
> > >
> > > --
> > > Nicolò Boschi
> > >
> >
>

Re: Dropping Presto SQL in 2.9.0 - status ?

Posted by Zhengxin Cai <ca...@gmail.com>.

Hi there,
I think the pr is still open, https://github.com/trinodb/trino/pull/8020,
will try to push it.
But even after the pr is merged, I actually still think we might still want
to keep a copy of the connector in Pulsar repo and push changes to Trino
repo periodically, as this will allow much faster bug fix and feature
iteration.
Best,
Marvin,

Lari Hotari <lh...@apache.org> 于2021年11月17日周三 下午2:19写道：

> Dear Pulsar community members,
>
> PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
> repositories" was created in April 2020. The repositories for
> pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
> ago [2].
>
> What is the current roadmap for completing PIP-62 and moving
> pulsar-connectors and pulsar-sql out of apache/pulsar repository?
>
> BR,
>
> Lari
>
> [1]
>
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> [2]
>
> https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E
>
> Other email threads:
> * [Discuss] Don't include presto/trino in the normal Pulsar distribution -
> https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
> * Updates on Presto connector for PIP-62 -
> https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk
>
> On Tue, Nov 2, 2021 at 3:59 PM Nicolò Boschi <bo...@gmail.com> wrote:
>
> > Resurrecting this thread.
> >
> > 2.9 is almost released and it hasn't been merged yet
> >
> > Extending the discussion to other connectors, it looks like there has
> been
> > no progress on PIP-62.
> > My concern is that a lot of Pulsar IO connectors dependencies we are
> > running are obsolete with several security reports
> >
> > I see there are interesting comments in the issue (
> > https://github.com/apache/pulsar/issues/10219) and Sijie exported the
> > pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
> > outdated
> >
> > From my point of view, we have to:
> > - reimport all the connectors source codes with newest ones (including
> > integration tests)
> > - add periodic CI jobs for connectors to run against master, 2.9-latest,
> > 2.8-latest, 2.7-latest to verify breaking changes
> > - define a release cycle/management for connectors (we should improve the
> > PIP doc). IMO it's not clear if each connector will have its own release
> > versions and how we'll handle it (git tags, artifacts deployment..)
> > - update pulsar release script in order to get the connectors artifacts
> > (retrieving the .nar or building it from source?)
> > - update docs
> > - remove pulsar-io dir from Pulsar repo
> >
> > It's the perfect timing to schedule this work for 2.10
> >
> > What is missing? How's the situation? Is there a roadblock I haven't
> seen?
> > I think it's better to take another discussion for Presto since it will
> > come to another end
> >
> >
> > Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
> > eolivelli@gmail.com> ha scritto:
> >
> > > Sijie
> > >
> > > Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
> > >
> > > > You can follow the progress at
> > > https://github.com/trinodb/trino/pull/8020.
> > > >
> > >
> > > Thanks for the pointer
> > >
> > > >
> > > > The original code doesn't conform to TrinoDB's standard. Marvin is
> > > > actively following up on that.
> > > >
> > > > Our goal is still to get this completed as part of the 2.9 release.
> > > >
> > >
> > > Wonderful
> > >
> > > Thanks
> > > Enrico
> > >
> > > >
> > > > - Sijie
> > > >
> > > > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <eolivelli@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Hello,
> > > > > How is the Presto work going ?
> > > > > IIRC the plan was to remove it from the Pulsar code base and let it
> > be
> > > > > hosted at Trino.
> > > > >
> > > > > If this is not going to happen within the 2.9.0 release timeline
> > > > > (September?) I would prefer to upgrade to "Trino".
> > > > > Probably we will have a downside problem that recent versions of
> > > > > Presto/Trino do not work on JDK8 but only on JDK11.
> > > > >
> > > > > I believe that in that case we could open a separate thread to say
> > that
> > > > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and it
> is
> > > the
> > > > > preferred version, as it is the version used in the Docker images),
> > so
> > > > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > > > >
> > > > > My primary concern is that the version of Presto that we are
> running
> > is
> > > > > obsolete and there are several security reports against it or its
> > third
> > > > > party dependencies.
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > Enrico
> > > >
> > >
> >
> >
> > --
> > Nicolò Boschi
> >
>

Re: Dropping Presto SQL in 2.9.0 - status ?

Posted by Lari Hotari <lh...@apache.org>.

Dear Pulsar community members,

PIP-62[1], "PIP 62: Move connectors, adapters and Pulsar Presto to separate
repositories" was created in April 2020. The repositories for
pulsar-connectors, pulsar-adapters and pulsar-sql were created about a year
ago [2].

What is the current roadmap for completing PIP-62 and moving
pulsar-connectors and pulsar-sql out of apache/pulsar repository?

BR,

Lari

[1]
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
[2]
https://lists.apache.org/thread.html/r9e6ec742e2896da1f0ce7d4adc7cb84fc6db6dbf797732ccdd50fb86%40%3Cdev.pulsar.apache.org%3E

Other email threads:
* [Discuss] Don't include presto/trino in the normal Pulsar distribution -
https://lists.apache.org/thread/jn96tct54mn0tvdot62vdslrvs38fm6d
* Updates on Presto connector for PIP-62 -
https://lists.apache.org/thread/f9n6sc2mrboq5sxhjbr7gvdl8vqp9fpk

On Tue, Nov 2, 2021 at 3:59 PM Nicolò Boschi <bo...@gmail.com> wrote:

> Resurrecting this thread.
>
> 2.9 is almost released and it hasn't been merged yet
>
> Extending the discussion to other connectors, it looks like there has been
> no progress on PIP-62.
> My concern is that a lot of Pulsar IO connectors dependencies we are
> running are obsolete with several security reports
>
> I see there are interesting comments in the issue (
> https://github.com/apache/pulsar/issues/10219) and Sijie exported the
> pulsar-io dir to https://github.com/apache/pulsar-connectors but it's
> outdated
>
> From my point of view, we have to:
> - reimport all the connectors source codes with newest ones (including
> integration tests)
> - add periodic CI jobs for connectors to run against master, 2.9-latest,
> 2.8-latest, 2.7-latest to verify breaking changes
> - define a release cycle/management for connectors (we should improve the
> PIP doc). IMO it's not clear if each connector will have its own release
> versions and how we'll handle it (git tags, artifacts deployment..)
> - update pulsar release script in order to get the connectors artifacts
> (retrieving the .nar or building it from source?)
> - update docs
> - remove pulsar-io dir from Pulsar repo
>
> It's the perfect timing to schedule this work for 2.10
>
> What is missing? How's the situation? Is there a roadblock I haven't seen?
> I think it's better to take another discussion for Presto since it will
> come to another end
>
>
> Il giorno sab 14 ago 2021 alle ore 15:21 Enrico Olivelli <
> eolivelli@gmail.com> ha scritto:
>
> > Sijie
> >
> > Il Ven 13 Ago 2021, 22:00 Sijie Guo <gu...@gmail.com> ha scritto:
> >
> > > You can follow the progress at
> > https://github.com/trinodb/trino/pull/8020.
> > >
> >
> > Thanks for the pointer
> >
> > >
> > > The original code doesn't conform to TrinoDB's standard. Marvin is
> > > actively following up on that.
> > >
> > > Our goal is still to get this completed as part of the 2.9 release.
> > >
> >
> > Wonderful
> >
> > Thanks
> > Enrico
> >
> > >
> > > - Sijie
> > >
> > > On Fri, Aug 13, 2021 at 2:04 AM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > > >
> > > > Hello,
> > > > How is the Presto work going ?
> > > > IIRC the plan was to remove it from the Pulsar code base and let it
> be
> > > > hosted at Trino.
> > > >
> > > > If this is not going to happen within the 2.9.0 release timeline
> > > > (September?) I would prefer to upgrade to "Trino".
> > > > Probably we will have a downside problem that recent versions of
> > > > Presto/Trino do not work on JDK8 but only on JDK11.
> > > >
> > > > I believe that in that case we could open a separate thread to say
> that
> > > > Pulsar SQL in 2.9.0 will work only on JDK11.
> > > > In Pulsar 2.8.0 we added official compatibility with JDK11 (and it is
> > the
> > > > preferred version, as it is the version used in the Docker images),
> so
> > > > requiring JDK11 for Pulsar SQL 2.9.0 does not sound bad to me.
> > > >
> > > > My primary concern is that the version of Presto that we are running
> is
> > > > obsolete and there are several security reports against it or its
> third
> > > > party dependencies.
> > > >
> > > > Thoughts ?
> > > >
> > > > Enrico
> > >
> >
>
>
> --
> Nicolò Boschi
>