You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Matteo Merli <mm...@apache.org> on 2023/12/22 17:09:19 UTC
[DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
I want to start a discussion regarding the removal of all the code related
to the Trino (PrestoDB) plugin from the Pulsar main repository.
This topic was already discussed and approved long time ago in PIP-62 (
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
)
The main reasons for not having Presto plugin as part of the main
distribution of Pulsar were (and still are valid):
1. We need to ship the entire Presto runtime which is ~400 MB. This makes
our tgz and Docker images huge
3. There is no strict need for this component to be in the same
distribution / image: it could easily be provided in a different release
tgz or Docker image
Though I think that since then it became more clear that the current state
of this plugin has been stagnating over the years.
1. There are not many active users of Pulsar-SQL component (I'd be very
happy to be contradicted here)
2. The plugin code has not been improved in a long time
3. There are several open security issues (actually, almost the totality of
current dependencies issues are today coming from Trino).
My suggestion would be that, if there is any volunteer willing to pick this
plugin up and maintain it in a separate repository (within the Apache
Pulsar project) and with a separate release schedule, we should go ahead
and move it.
If there are no volunteers, we should just remove it as it is. If later on
we want to revive it, we can always import the code from the last commit.
Thoughts?
--
Matteo Merli
<mm...@apache.org>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Matteo Merli <ma...@gmail.com>.
Since there seems to be agreement here, I'll proceed with the PR for
removing the code.
If there are volunteers that are willing to continue maintaining the Trino
plugin, I'll be happy to help out in setting up the new repository and
import the existing code.
Thanks,
Matteo
--
Matteo Merli
<ma...@gmail.com>
On Sun, Dec 24, 2023 at 7:47 PM Yubiao Feng
<yu...@streamnative.io.invalid> wrote:
> +1
>
> Thanks
> Yubiao Feng
>
> On Sat, Dec 23, 2023 at 1:09 AM Matteo Merli <mm...@apache.org> wrote:
>
> > I want to start a discussion regarding the removal of all the code
> related
> > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> >
> > This topic was already discussed and approved long time ago in PIP-62 (
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > )
> >
> > The main reasons for not having Presto plugin as part of the main
> > distribution of Pulsar were (and still are valid):
> >
> > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> makes
> > our tgz and Docker images huge
> > 3. There is no strict need for this component to be in the same
> > distribution / image: it could easily be provided in a different release
> > tgz or Docker image
> >
> > Though I think that since then it became more clear that the current
> state
> > of this plugin has been stagnating over the years.
> >
> > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > happy to be contradicted here)
> > 2. The plugin code has not been improved in a long time
> > 3. There are several open security issues (actually, almost the totality
> of
> > current dependencies issues are today coming from Trino).
> >
> > My suggestion would be that, if there is any volunteer willing to pick
> this
> > plugin up and maintain it in a separate repository (within the Apache
> > Pulsar project) and with a separate release schedule, we should go ahead
> > and move it.
> > If there are no volunteers, we should just remove it as it is. If later
> on
> > we want to revive it, we can always import the code from the last commit.
> >
> > Thoughts?
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Yubiao Feng <yu...@streamnative.io.INVALID>.
+1
Thanks
Yubiao Feng
On Sat, Dec 23, 2023 at 1:09 AM Matteo Merli <mm...@apache.org> wrote:
> I want to start a discussion regarding the removal of all the code related
> to the Trino (PrestoDB) plugin from the Pulsar main repository.
>
> This topic was already discussed and approved long time ago in PIP-62 (
>
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> )
>
> The main reasons for not having Presto plugin as part of the main
> distribution of Pulsar were (and still are valid):
>
> 1. We need to ship the entire Presto runtime which is ~400 MB. This makes
> our tgz and Docker images huge
> 3. There is no strict need for this component to be in the same
> distribution / image: it could easily be provided in a different release
> tgz or Docker image
>
> Though I think that since then it became more clear that the current state
> of this plugin has been stagnating over the years.
>
> 1. There are not many active users of Pulsar-SQL component (I'd be very
> happy to be contradicted here)
> 2. The plugin code has not been improved in a long time
> 3. There are several open security issues (actually, almost the totality of
> current dependencies issues are today coming from Trino).
>
> My suggestion would be that, if there is any volunteer willing to pick this
> plugin up and maintain it in a separate repository (within the Apache
> Pulsar project) and with a separate release schedule, we should go ahead
> and move it.
> If there are no volunteers, we should just remove it as it is. If later on
> we want to revive it, we can always import the code from the last commit.
>
> Thoughts?
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by PengHui Li <pe...@apache.org>.
+1
Regards,
Penghui
On Mon, Dec 25, 2023 at 10:06 AM guo jiwei <te...@apache.org> wrote:
> yes, +1 agree
>
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Sat, Dec 23, 2023 at 11:37 AM Matteo Merli <ma...@gmail.com>
> wrote:
>
> > Good point. I have created a PR here:
> > https://github.com/apache/pulsar/pull/21795
> >
> > We can always cherry-pick changes into the release branch, during the
> code
> > freeze, with the appropriate precautions :)
> >
> > Matteo
> >
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> >
> > On Fri, Dec 22, 2023 at 9:41 AM Lari Hotari <lh...@apache.org> wrote:
> >
> > > Do we target release 3.2.0 if this change gets accepted?
> > >
> > > The code freeze for 3.2.0 was about to start today and I blocked that.
> I
> > > removed the already started branch-3.2 (which didn't contain any
> changes)
> > > until the decision has been made. I hope this makes sense. If not, we
> can
> > > always cut the branch again, nothing has been lost.
> > >
> > > re: https://lists.apache.org/thread/0pl9by5c53nkjp7vld3x89c8nqjrx9on
> > >
> > > -Lari
> > >
> > > On 2023/12/22 17:09:19 Matteo Merli wrote:
> > > > I want to start a discussion regarding the removal of all the code
> > > related
> > > > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> > > >
> > > > This topic was already discussed and approved long time ago in
> PIP-62 (
> > > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > > )
> > > >
> > > > The main reasons for not having Presto plugin as part of the main
> > > > distribution of Pulsar were (and still are valid):
> > > >
> > > > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> > > makes
> > > > our tgz and Docker images huge
> > > > 3. There is no strict need for this component to be in the same
> > > > distribution / image: it could easily be provided in a different
> > release
> > > > tgz or Docker image
> > > >
> > > > Though I think that since then it became more clear that the current
> > > state
> > > > of this plugin has been stagnating over the years.
> > > >
> > > > 1. There are not many active users of Pulsar-SQL component (I'd be
> very
> > > > happy to be contradicted here)
> > > > 2. The plugin code has not been improved in a long time
> > > > 3. There are several open security issues (actually, almost the
> > totality
> > > of
> > > > current dependencies issues are today coming from Trino).
> > > >
> > > > My suggestion would be that, if there is any volunteer willing to
> pick
> > > this
> > > > plugin up and maintain it in a separate repository (within the Apache
> > > > Pulsar project) and with a separate release schedule, we should go
> > ahead
> > > > and move it.
> > > > If there are no volunteers, we should just remove it as it is. If
> later
> > > on
> > > > we want to revive it, we can always import the code from the last
> > commit.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > <mm...@apache.org>
> > > >
> > >
> >
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by guo jiwei <te...@apache.org>.
yes, +1 agree
Regards
Jiwei Guo (Tboy)
On Sat, Dec 23, 2023 at 11:37 AM Matteo Merli <ma...@gmail.com>
wrote:
> Good point. I have created a PR here:
> https://github.com/apache/pulsar/pull/21795
>
> We can always cherry-pick changes into the release branch, during the code
> freeze, with the appropriate precautions :)
>
> Matteo
>
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>
> On Fri, Dec 22, 2023 at 9:41 AM Lari Hotari <lh...@apache.org> wrote:
>
> > Do we target release 3.2.0 if this change gets accepted?
> >
> > The code freeze for 3.2.0 was about to start today and I blocked that. I
> > removed the already started branch-3.2 (which didn't contain any changes)
> > until the decision has been made. I hope this makes sense. If not, we can
> > always cut the branch again, nothing has been lost.
> >
> > re: https://lists.apache.org/thread/0pl9by5c53nkjp7vld3x89c8nqjrx9on
> >
> > -Lari
> >
> > On 2023/12/22 17:09:19 Matteo Merli wrote:
> > > I want to start a discussion regarding the removal of all the code
> > related
> > > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> > >
> > > This topic was already discussed and approved long time ago in PIP-62 (
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > )
> > >
> > > The main reasons for not having Presto plugin as part of the main
> > > distribution of Pulsar were (and still are valid):
> > >
> > > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> > makes
> > > our tgz and Docker images huge
> > > 3. There is no strict need for this component to be in the same
> > > distribution / image: it could easily be provided in a different
> release
> > > tgz or Docker image
> > >
> > > Though I think that since then it became more clear that the current
> > state
> > > of this plugin has been stagnating over the years.
> > >
> > > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > > happy to be contradicted here)
> > > 2. The plugin code has not been improved in a long time
> > > 3. There are several open security issues (actually, almost the
> totality
> > of
> > > current dependencies issues are today coming from Trino).
> > >
> > > My suggestion would be that, if there is any volunteer willing to pick
> > this
> > > plugin up and maintain it in a separate repository (within the Apache
> > > Pulsar project) and with a separate release schedule, we should go
> ahead
> > > and move it.
> > > If there are no volunteers, we should just remove it as it is. If later
> > on
> > > we want to revive it, we can always import the code from the last
> commit.
> > >
> > > Thoughts?
> > >
> > >
> > > --
> > > Matteo Merli
> > > <mm...@apache.org>
> > >
> >
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Matteo Merli <ma...@gmail.com>.
Good point. I have created a PR here:
https://github.com/apache/pulsar/pull/21795
We can always cherry-pick changes into the release branch, during the code
freeze, with the appropriate precautions :)
Matteo
--
Matteo Merli
<ma...@gmail.com>
On Fri, Dec 22, 2023 at 9:41 AM Lari Hotari <lh...@apache.org> wrote:
> Do we target release 3.2.0 if this change gets accepted?
>
> The code freeze for 3.2.0 was about to start today and I blocked that. I
> removed the already started branch-3.2 (which didn't contain any changes)
> until the decision has been made. I hope this makes sense. If not, we can
> always cut the branch again, nothing has been lost.
>
> re: https://lists.apache.org/thread/0pl9by5c53nkjp7vld3x89c8nqjrx9on
>
> -Lari
>
> On 2023/12/22 17:09:19 Matteo Merli wrote:
> > I want to start a discussion regarding the removal of all the code
> related
> > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> >
> > This topic was already discussed and approved long time ago in PIP-62 (
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > )
> >
> > The main reasons for not having Presto plugin as part of the main
> > distribution of Pulsar were (and still are valid):
> >
> > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> makes
> > our tgz and Docker images huge
> > 3. There is no strict need for this component to be in the same
> > distribution / image: it could easily be provided in a different release
> > tgz or Docker image
> >
> > Though I think that since then it became more clear that the current
> state
> > of this plugin has been stagnating over the years.
> >
> > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > happy to be contradicted here)
> > 2. The plugin code has not been improved in a long time
> > 3. There are several open security issues (actually, almost the totality
> of
> > current dependencies issues are today coming from Trino).
> >
> > My suggestion would be that, if there is any volunteer willing to pick
> this
> > plugin up and maintain it in a separate repository (within the Apache
> > Pulsar project) and with a separate release schedule, we should go ahead
> > and move it.
> > If there are no volunteers, we should just remove it as it is. If later
> on
> > we want to revive it, we can always import the code from the last commit.
> >
> > Thoughts?
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Lari Hotari <lh...@apache.org>.
Do we target release 3.2.0 if this change gets accepted?
The code freeze for 3.2.0 was about to start today and I blocked that. I removed the already started branch-3.2 (which didn't contain any changes)
until the decision has been made. I hope this makes sense. If not, we can always cut the branch again, nothing has been lost.
re: https://lists.apache.org/thread/0pl9by5c53nkjp7vld3x89c8nqjrx9on
-Lari
On 2023/12/22 17:09:19 Matteo Merli wrote:
> I want to start a discussion regarding the removal of all the code related
> to the Trino (PrestoDB) plugin from the Pulsar main repository.
>
> This topic was already discussed and approved long time ago in PIP-62 (
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> )
>
> The main reasons for not having Presto plugin as part of the main
> distribution of Pulsar were (and still are valid):
>
> 1. We need to ship the entire Presto runtime which is ~400 MB. This makes
> our tgz and Docker images huge
> 3. There is no strict need for this component to be in the same
> distribution / image: it could easily be provided in a different release
> tgz or Docker image
>
> Though I think that since then it became more clear that the current state
> of this plugin has been stagnating over the years.
>
> 1. There are not many active users of Pulsar-SQL component (I'd be very
> happy to be contradicted here)
> 2. The plugin code has not been improved in a long time
> 3. There are several open security issues (actually, almost the totality of
> current dependencies issues are today coming from Trino).
>
> My suggestion would be that, if there is any volunteer willing to pick this
> plugin up and maintain it in a separate repository (within the Apache
> Pulsar project) and with a separate release schedule, we should go ahead
> and move it.
> If there are no volunteers, we should just remove it as it is. If later on
> we want to revive it, we can always import the code from the last commit.
>
> Thoughts?
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Lari Hotari <lh...@apache.org>.
Good plan!
-Lari
On Fri, 22 Dec 2023 at 19:09, Matteo Merli <mm...@apache.org> wrote:
>
> I want to start a discussion regarding the removal of all the code related
> to the Trino (PrestoDB) plugin from the Pulsar main repository.
>
> This topic was already discussed and approved long time ago in PIP-62 (
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> )
>
> The main reasons for not having Presto plugin as part of the main
> distribution of Pulsar were (and still are valid):
>
> 1. We need to ship the entire Presto runtime which is ~400 MB. This makes
> our tgz and Docker images huge
> 3. There is no strict need for this component to be in the same
> distribution / image: it could easily be provided in a different release
> tgz or Docker image
>
> Though I think that since then it became more clear that the current state
> of this plugin has been stagnating over the years.
>
> 1. There are not many active users of Pulsar-SQL component (I'd be very
> happy to be contradicted here)
> 2. The plugin code has not been improved in a long time
> 3. There are several open security issues (actually, almost the totality of
> current dependencies issues are today coming from Trino).
>
> My suggestion would be that, if there is any volunteer willing to pick this
> plugin up and maintain it in a separate repository (within the Apache
> Pulsar project) and with a separate release schedule, we should go ahead
> and move it.
> If there are no volunteers, we should just remove it as it is. If later on
> we want to revive it, we can always import the code from the last commit.
>
> Thoughts?
>
>
> --
> Matteo Merli
> <mm...@apache.org>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Lari Hotari <lh...@apache.org>.
Thank you! Those look like valuable changes. Yes, copying over the PRs
once there's a build and CI setup in apache/pulsar-sql would be great.
It's worth postponing until
https://github.com/apache/pulsar-sql/issues/3 has been resolved.
-Lari
On Fri, 18 Oct 2024 at 11:45, Jie crossover <cr...@gmail.com> wrote:
>
> Hi lari,
>
> I still have some pending PRs, can I just copy them over?
>
> https://github.com/apache/pulsar/pulls?q=sort%3Aupdated-desc+is%3Apr+is%3Aopen+author%3AcrossoverJie+sql
> --
> Best Regards！
> crossoverJie
>
>
> Zixuan Liu <no...@gmail.com> 于2024年10月18日周五 16:20写道：
>
> > Hi lari,
> >
> > I can drive the build system and CI setup, could you assign this task to
> > me?
> >
> > Thanks,
> > Zixuan
> >
> > Lari Hotari <lh...@apache.org> 于2024年10月18日周五 13:51写道：
> >
> > > There has been some interest in continuing to use Pulsar SQL in the
> > > future. To facilitate this, I have taken the following actions:
> > > - Extracted the pulsar-sql files from branch-3.0 to a new repository:
> > > https://github.com/apache/pulsar-sql
> > > - This new repository allows interested volunteers to make contributions,
> > > which has been a barrier until now.
> > > - The README file in https://github.com/apache/pulsar-sql explains how
> > > the pulsar-sql files were extracted with commit history to the new
> > > repository.
> > >
> > > Please note:
> > > - The new repository currently lacks a build system and CI setup.
> > > - Setting up the build and CI will be the first task for volunteers
> > > interested in maintaining this component.
> > >
> > > We welcome contributions from the community to help maintain and improve
> > > Pulsar SQL going forward. In the meantime, current Pulsar SQL users can
> > > continue to use Pulsar 3.0.x LTS, which is supported until May 2025.
> > There
> > > are more details in the GitHub Discussions answer,
> > > https://github.com/apache/pulsar/discussions/23435
> > >
> > > -Lari
> > >
> > > On 2023/12/22 17:09:19 Matteo Merli wrote:
> > > > I want to start a discussion regarding the removal of all the code
> > > related
> > > > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> > > >
> > > > This topic was already discussed and approved long time ago in PIP-62 (
> > > >
> > >
> > https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > > )
> > > >
> > > > The main reasons for not having Presto plugin as part of the main
> > > > distribution of Pulsar were (and still are valid):
> > > >
> > > > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> > > makes
> > > > our tgz and Docker images huge
> > > > 3. There is no strict need for this component to be in the same
> > > > distribution / image: it could easily be provided in a different
> > release
> > > > tgz or Docker image
> > > >
> > > > Though I think that since then it became more clear that the current
> > > state
> > > > of this plugin has been stagnating over the years.
> > > >
> > > > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > > > happy to be contradicted here)
> > > > 2. The plugin code has not been improved in a long time
> > > > 3. There are several open security issues (actually, almost the
> > totality
> > > of
> > > > current dependencies issues are today coming from Trino).
> > > >
> > > > My suggestion would be that, if there is any volunteer willing to pick
> > > this
> > > > plugin up and maintain it in a separate repository (within the Apache
> > > > Pulsar project) and with a separate release schedule, we should go
> > ahead
> > > > and move it.
> > > > If there are no volunteers, we should just remove it as it is. If later
> > > on
> > > > we want to revive it, we can always import the code from the last
> > commit.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > <mm...@apache.org>
> > > >
> > >
> >
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Jie crossover <cr...@gmail.com>.
Hi lari,
I still have some pending PRs, can I just copy them over?
https://github.com/apache/pulsar/pulls?q=sort%3Aupdated-desc+is%3Apr+is%3Aopen+author%3AcrossoverJie+sql
--
Best Regards！
crossoverJie
Zixuan Liu <no...@gmail.com> 于2024年10月18日周五 16:20写道：
> Hi lari,
>
> I can drive the build system and CI setup, could you assign this task to
> me?
>
> Thanks,
> Zixuan
>
> Lari Hotari <lh...@apache.org> 于2024年10月18日周五 13:51写道：
>
> > There has been some interest in continuing to use Pulsar SQL in the
> > future. To facilitate this, I have taken the following actions:
> > - Extracted the pulsar-sql files from branch-3.0 to a new repository:
> > https://github.com/apache/pulsar-sql
> > - This new repository allows interested volunteers to make contributions,
> > which has been a barrier until now.
> > - The README file in https://github.com/apache/pulsar-sql explains how
> > the pulsar-sql files were extracted with commit history to the new
> > repository.
> >
> > Please note:
> > - The new repository currently lacks a build system and CI setup.
> > - Setting up the build and CI will be the first task for volunteers
> > interested in maintaining this component.
> >
> > We welcome contributions from the community to help maintain and improve
> > Pulsar SQL going forward. In the meantime, current Pulsar SQL users can
> > continue to use Pulsar 3.0.x LTS, which is supported until May 2025.
> There
> > are more details in the GitHub Discussions answer,
> > https://github.com/apache/pulsar/discussions/23435
> >
> > -Lari
> >
> > On 2023/12/22 17:09:19 Matteo Merli wrote:
> > > I want to start a discussion regarding the removal of all the code
> > related
> > > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> > >
> > > This topic was already discussed and approved long time ago in PIP-62 (
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > )
> > >
> > > The main reasons for not having Presto plugin as part of the main
> > > distribution of Pulsar were (and still are valid):
> > >
> > > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> > makes
> > > our tgz and Docker images huge
> > > 3. There is no strict need for this component to be in the same
> > > distribution / image: it could easily be provided in a different
> release
> > > tgz or Docker image
> > >
> > > Though I think that since then it became more clear that the current
> > state
> > > of this plugin has been stagnating over the years.
> > >
> > > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > > happy to be contradicted here)
> > > 2. The plugin code has not been improved in a long time
> > > 3. There are several open security issues (actually, almost the
> totality
> > of
> > > current dependencies issues are today coming from Trino).
> > >
> > > My suggestion would be that, if there is any volunteer willing to pick
> > this
> > > plugin up and maintain it in a separate repository (within the Apache
> > > Pulsar project) and with a separate release schedule, we should go
> ahead
> > > and move it.
> > > If there are no volunteers, we should just remove it as it is. If later
> > on
> > > we want to revive it, we can always import the code from the last
> commit.
> > >
> > > Thoughts?
> > >
> > >
> > > --
> > > Matteo Merli
> > > <mm...@apache.org>
> > >
> >
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Lari Hotari <La...@hotari.net>.
Thanks for volunteering! I created
https://github.com/apache/pulsar-sql/issues/3 and assigned it to you.
Please ask for further clarification directly on the issue. You can
also reach out to me on the #dev channel in Pulsar slack.
For version numbering, I think that we could start with
4.0.0-SNAPSHOT. The first release would be 4.0.0. The releases would
be independent from apache/pulsar releases. That could be confusing,
but that could be explained in the README.
-Lari
On Fri, 18 Oct 2024 at 11:21, Zixuan Liu <no...@gmail.com> wrote:
>
> Hi lari,
>
> I can drive the build system and CI setup, could you assign this task to me?
>
> Thanks,
> Zixuan
>
> Lari Hotari <lh...@apache.org> 于2024年10月18日周五 13:51写道：
>
> > There has been some interest in continuing to use Pulsar SQL in the
> > future. To facilitate this, I have taken the following actions:
> > - Extracted the pulsar-sql files from branch-3.0 to a new repository:
> > https://github.com/apache/pulsar-sql
> > - This new repository allows interested volunteers to make contributions,
> > which has been a barrier until now.
> > - The README file in https://github.com/apache/pulsar-sql explains how
> > the pulsar-sql files were extracted with commit history to the new
> > repository.
> >
> > Please note:
> > - The new repository currently lacks a build system and CI setup.
> > - Setting up the build and CI will be the first task for volunteers
> > interested in maintaining this component.
> >
> > We welcome contributions from the community to help maintain and improve
> > Pulsar SQL going forward. In the meantime, current Pulsar SQL users can
> > continue to use Pulsar 3.0.x LTS, which is supported until May 2025. There
> > are more details in the GitHub Discussions answer,
> > https://github.com/apache/pulsar/discussions/23435
> >
> > -Lari
> >
> > On 2023/12/22 17:09:19 Matteo Merli wrote:
> > > I want to start a discussion regarding the removal of all the code
> > related
> > > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> > >
> > > This topic was already discussed and approved long time ago in PIP-62 (
> > >
> > https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > > )
> > >
> > > The main reasons for not having Presto plugin as part of the main
> > > distribution of Pulsar were (and still are valid):
> > >
> > > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> > makes
> > > our tgz and Docker images huge
> > > 3. There is no strict need for this component to be in the same
> > > distribution / image: it could easily be provided in a different release
> > > tgz or Docker image
> > >
> > > Though I think that since then it became more clear that the current
> > state
> > > of this plugin has been stagnating over the years.
> > >
> > > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > > happy to be contradicted here)
> > > 2. The plugin code has not been improved in a long time
> > > 3. There are several open security issues (actually, almost the totality
> > of
> > > current dependencies issues are today coming from Trino).
> > >
> > > My suggestion would be that, if there is any volunteer willing to pick
> > this
> > > plugin up and maintain it in a separate repository (within the Apache
> > > Pulsar project) and with a separate release schedule, we should go ahead
> > > and move it.
> > > If there are no volunteers, we should just remove it as it is. If later
> > on
> > > we want to revive it, we can always import the code from the last commit.
> > >
> > > Thoughts?
> > >
> > >
> > > --
> > > Matteo Merli
> > > <mm...@apache.org>
> > >
> >
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Zixuan Liu <no...@gmail.com>.
Hi lari,
I can drive the build system and CI setup, could you assign this task to me?
Thanks,
Zixuan
Lari Hotari <lh...@apache.org> 于2024年10月18日周五 13:51写道：
> There has been some interest in continuing to use Pulsar SQL in the
> future. To facilitate this, I have taken the following actions:
> - Extracted the pulsar-sql files from branch-3.0 to a new repository:
> https://github.com/apache/pulsar-sql
> - This new repository allows interested volunteers to make contributions,
> which has been a barrier until now.
> - The README file in https://github.com/apache/pulsar-sql explains how
> the pulsar-sql files were extracted with commit history to the new
> repository.
>
> Please note:
> - The new repository currently lacks a build system and CI setup.
> - Setting up the build and CI will be the first task for volunteers
> interested in maintaining this component.
>
> We welcome contributions from the community to help maintain and improve
> Pulsar SQL going forward. In the meantime, current Pulsar SQL users can
> continue to use Pulsar 3.0.x LTS, which is supported until May 2025. There
> are more details in the GitHub Discussions answer,
> https://github.com/apache/pulsar/discussions/23435
>
> -Lari
>
> On 2023/12/22 17:09:19 Matteo Merli wrote:
> > I want to start a discussion regarding the removal of all the code
> related
> > to the Trino (PrestoDB) plugin from the Pulsar main repository.
> >
> > This topic was already discussed and approved long time ago in PIP-62 (
> >
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> > )
> >
> > The main reasons for not having Presto plugin as part of the main
> > distribution of Pulsar were (and still are valid):
> >
> > 1. We need to ship the entire Presto runtime which is ~400 MB. This
> makes
> > our tgz and Docker images huge
> > 3. There is no strict need for this component to be in the same
> > distribution / image: it could easily be provided in a different release
> > tgz or Docker image
> >
> > Though I think that since then it became more clear that the current
> state
> > of this plugin has been stagnating over the years.
> >
> > 1. There are not many active users of Pulsar-SQL component (I'd be very
> > happy to be contradicted here)
> > 2. The plugin code has not been improved in a long time
> > 3. There are several open security issues (actually, almost the totality
> of
> > current dependencies issues are today coming from Trino).
> >
> > My suggestion would be that, if there is any volunteer willing to pick
> this
> > plugin up and maintain it in a separate repository (within the Apache
> > Pulsar project) and with a separate release schedule, we should go ahead
> > and move it.
> > If there are no volunteers, we should just remove it as it is. If later
> on
> > we want to revive it, we can always import the code from the last commit.
> >
> > Thoughts?
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
>
Re: [DISCUSS] Removing Pulsar-Trino plugin from main repo and call for volunteers to maintain it
Posted by Lari Hotari <lh...@apache.org>.
There has been some interest in continuing to use Pulsar SQL in the future. To facilitate this, I have taken the following actions:
- Extracted the pulsar-sql files from branch-3.0 to a new repository: https://github.com/apache/pulsar-sql
- This new repository allows interested volunteers to make contributions, which has been a barrier until now.
- The README file in https://github.com/apache/pulsar-sql explains how the pulsar-sql files were extracted with commit history to the new repository.
Please note:
- The new repository currently lacks a build system and CI setup.
- Setting up the build and CI will be the first task for volunteers interested in maintaining this component.
We welcome contributions from the community to help maintain and improve Pulsar SQL going forward. In the meantime, current Pulsar SQL users can continue to use Pulsar 3.0.x LTS, which is supported until May 2025. There are more details in the GitHub Discussions answer, https://github.com/apache/pulsar/discussions/23435
-Lari
On 2023/12/22 17:09:19 Matteo Merli wrote:
> I want to start a discussion regarding the removal of all the code related
> to the Trino (PrestoDB) plugin from the Pulsar main repository.
>
> This topic was already discussed and approved long time ago in PIP-62 (
> https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
> )
>
> The main reasons for not having Presto plugin as part of the main
> distribution of Pulsar were (and still are valid):
>
> 1. We need to ship the entire Presto runtime which is ~400 MB. This makes
> our tgz and Docker images huge
> 3. There is no strict need for this component to be in the same
> distribution / image: it could easily be provided in a different release
> tgz or Docker image
>
> Though I think that since then it became more clear that the current state
> of this plugin has been stagnating over the years.
>
> 1. There are not many active users of Pulsar-SQL component (I'd be very
> happy to be contradicted here)
> 2. The plugin code has not been improved in a long time
> 3. There are several open security issues (actually, almost the totality of
> current dependencies issues are today coming from Trino).
>
> My suggestion would be that, if there is any volunteer willing to pick this
> plugin up and maintain it in a separate repository (within the Apache
> Pulsar project) and with a separate release schedule, we should go ahead
> and move it.
> If there are no volunteers, we should just remove it as it is. If later on
> we want to revive it, we can always import the code from the last commit.
>
> Thoughts?
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>