You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by tison <wa...@gmail.com> on 2022/08/17 09:13:58 UTC
[DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Hello,
This is a PIP to package the Pulsar Trino distro and config in a dedicated
folder.
Link: https://github.com/apache/pulsar/issues/17137
Prototype: https://github.com/apache/pulsar/pull/17062
Below you can find the proposal (I will amend the GH issue while we discuss
it).
Best,
tison.
Motivation
========
After https://github.com/apache/pulsar/pull/16683 merged, we upgrade
PrestoSQL dependency in Pulsar SQL to the first several Trino version. To
handle the name change cases and gradually refactor Pulsar SQL as a
self-contained module so that we can move it into a standalone repository,
I find that there're three major issues to resolve.
1. Configs of Pulsar SQL go under the `conf/` folder and mix with other
Pulsar configs.
2. Pulsar Docker images (base and all) bundle Pulsar SQL.
3. Integration tests of Pulsar SQL are tightly coupled with the main repo
(test infra).
This proposal is aimed at resolving the first issue to package Pulsar Trino
distro and config in a dedicated folder; that is, to make it self-contained.
Goal
====
I have already prepared a draft to perform the changes as
https://github.com/apache/pulsar/pull/17062. Generally, we move the config
files under `PRESTO_HOME` and correspondingly update scripts.
In this way, all Trino distro artifacts are under the same home path, so
that we can later move it out as a whole.
This change should not affect those who use Pulsar with the entry point
script, but it changes the layout of the release artifact, so I'd prefer to
perform a PIP process.
Implementation
============
It's straightforward to inline in the "Goal" section.
However, the name of the folder (`presto` or `trino`) and the level of the
folder (`lib/presto/` or `trino/`) is open to discussion. I think both are
fine and will try `trino/` first.
To minimize unnecessary changes, I tend to keep the modules name
`pulsar-presto-xxx` as is.
Alternatives
=========
I don't make a completed proposal to resolve all three issues listed above.
Because I'm still unfamiliar with the latter two topics yet and I'd prefer
to implement these improvements one by one since they're naturally
independent. If I try to make a completed proposal at once, it's highly
possible I give up halfway.
Anything else?
===========
Previous discussion:
[DISCUSS] Move Pulsar SQL to a separated repository?
https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by tison <wa...@gmail.com>.
Here is the vote thread:
https://lists.apache.org/thread/9stw26rrp9hm6l2xd2p78gpogw9bywgc
You're welcome to vote on the thread.
Best,
tison.
tison <wa...@gmail.com> 于2022年8月26日周五 09:06写道：
> Thanks for your input!
>
> I update the prototype as described in the proposal:
> https://github.com/apache/pulsar/pull/17062
>
> Will initialize a vote for PIP-200 soon.
>
> Best,
> tison.
>
>
> Zhengxin Cai <ca...@gmail.com> 于2022年8月26日周五 08:36写道：
>
>> +1 on this,
>> I think helm chart should not be a big issue, we can just upgrade the
>> chart
>> and provide necessary doc on how to upgrade.
>>
>> tison <wa...@gmail.com> 于2022年8月24日周三 16:41写道：
>>
>> > Hi Enrico and Qiang,
>> >
>> > The issue description was updated. Picked here:
>> >
>> > > upgrade and downgrade doc
>> >
>> > This change should not affect those who use Pulsar with the entry point
>> > script, but it changes the layout of the release artifact.
>> >
>> > I'm going to write a release note about this change and also post it on
>> the
>> > Pulsar SQL overview page as a caveat. Draft here:
>> >
>> > # Caveat
>> >
>> > If you're upgrading Pulsar SQL from 2.11 or early, you should copy the
>> > related configs from `conf/presto` to `trino/conf`, and `lib/presto` to
>> > `trino`. If you're downgrading Pulsar SQL to 2.11 or early from 2.12, do
>> > verse visa.
>> >
>> > > Pulsar Helm Chart
>> >
>> > From what I understand, Pulsar Helm Chart is a wrapper of pulsar-all
>> docker
>> > image. I don't find any reference to presto/trino/sql effectively in the
>> > codebase, so I think it's currently transparent for its users and the
>> > caveat under Pulsar SQL overview page should be enough.
>> >
>> > Best,
>> > tison.
>> >
>> >
>> > tison <wa...@gmail.com> 于2022年8月17日周三 19:40写道：
>> >
>> > > Thanks for your feedback!
>> > >
>> > > 1. According to the upgrade and downgrade doc, I think the minimum
>> > > requirements are a release note. Describe the layout change and how
>> users
>> > > should move the folder. I'll elaborate on the issue and notify you
>> here.
>> > > 2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how
>> it
>> > > can be relevant in days. If someone who maintains the Chart can
>> provide
>> > > some input, it will help!
>> > > 3. "There are 3 issues". It's a description about the
>> > > background/motivation, while we handle the first issue in this PIP.
>> > > Although, subtasks can be divided into packaging changes, possible doc
>> > > changes, and possible Chart changes.
>> > >
>> > > Best,
>> > > tison.
>> > >
>> > >
>> > > Qiang Huang <qi...@gmail.com> 于2022年8月17日周三 19:25写道：
>> > >
>> > >> Looks good. I have two points:
>> > >> 1. It is necessary to supplement the upgrade and downgrade
>> documentation
>> > >> in
>> > >> Pulsar.
>> > >> 2. There are 3 issues mentioned in the PIP, should we split it into 3
>> > >> small
>> > >> issues?
>> > >>
>> > >> Enrico Olivelli <eo...@gmail.com> 于2022年8月17日周三 17:30写道：
>> > >>
>> > >> > I generally agree with the PIP
>> > >> >
>> > >> > Can you please explain the interactions with the Pulsar Helm chart
>> ?
>> > >> > also we have to draw a migration path, because users that will
>> upgrade
>> > >> > Pulsar will have to move the configuration files in another
>> location
>> > >> >
>> > >> > Enrico
>> > >> >
>> > >> > Il giorno mer 17 ago 2022 alle ore 11:15 tison <
>> wander4096@gmail.com>
>> > >> > ha scritto:
>> > >> > >
>> > >> > > Hello,
>> > >> > >
>> > >> > > This is a PIP to package the Pulsar Trino distro and config in a
>> > >> > dedicated
>> > >> > > folder.
>> > >> > >
>> > >> > > Link: https://github.com/apache/pulsar/issues/17137
>> > >> > > Prototype: https://github.com/apache/pulsar/pull/17062
>> > >> > >
>> > >> > > Below you can find the proposal (I will amend the GH issue while
>> we
>> > >> > discuss
>> > >> > > it).
>> > >> > >
>> > >> > > Best,
>> > >> > > tison.
>> > >> > >
>> > >> > > Motivation
>> > >> > > ========
>> > >> > >
>> > >> > > After https://github.com/apache/pulsar/pull/16683 merged, we
>> > upgrade
>> > >> > > PrestoSQL dependency in Pulsar SQL to the first several Trino
>> > >> version. To
>> > >> > > handle the name change cases and gradually refactor Pulsar SQL
>> as a
>> > >> > > self-contained module so that we can move it into a standalone
>> > >> > repository,
>> > >> > > I find that there're three major issues to resolve.
>> > >> > >
>> > >> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with
>> > >> other
>> > >> > > Pulsar configs.
>> > >> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
>> > >> > > 3. Integration tests of Pulsar SQL are tightly coupled with the
>> main
>> > >> repo
>> > >> > > (test infra).
>> > >> > >
>> > >> > > This proposal is aimed at resolving the first issue to package
>> > Pulsar
>> > >> > Trino
>> > >> > > distro and config in a dedicated folder; that is, to make it
>> > >> > self-contained.
>> > >> > >
>> > >> > > Goal
>> > >> > > ====
>> > >> > >
>> > >> > > I have already prepared a draft to perform the changes as
>> > >> > > https://github.com/apache/pulsar/pull/17062. Generally, we move
>> the
>> > >> > config
>> > >> > > files under `PRESTO_HOME` and correspondingly update scripts.
>> > >> > >
>> > >> > > In this way, all Trino distro artifacts are under the same home
>> > path,
>> > >> so
>> > >> > > that we can later move it out as a whole.
>> > >> > >
>> > >> > > This change should not affect those who use Pulsar with the entry
>> > >> point
>> > >> > > script, but it changes the layout of the release artifact, so I'd
>> > >> prefer
>> > >> > to
>> > >> > > perform a PIP process.
>> > >> > >
>> > >> > > Implementation
>> > >> > > ============
>> > >> > >
>> > >> > > It's straightforward to inline in the "Goal" section.
>> > >> > >
>> > >> > > However, the name of the folder (`presto` or `trino`) and the
>> level
>> > of
>> > >> > the
>> > >> > > folder (`lib/presto/` or `trino/`) is open to discussion. I think
>> > both
>> > >> > are
>> > >> > > fine and will try `trino/` first.
>> > >> > >
>> > >> > > To minimize unnecessary changes, I tend to keep the modules name
>> > >> > > `pulsar-presto-xxx` as is.
>> > >> > >
>> > >> > > Alternatives
>> > >> > > =========
>> > >> > >
>> > >> > > I don't make a completed proposal to resolve all three issues
>> listed
>> > >> > above.
>> > >> > > Because I'm still unfamiliar with the latter two topics yet and
>> I'd
>> > >> > prefer
>> > >> > > to implement these improvements one by one since they're
>> naturally
>> > >> > > independent. If I try to make a completed proposal at once, it's
>> > >> highly
>> > >> > > possible I give up halfway.
>> > >> > >
>> > >> > > Anything else?
>> > >> > > ===========
>> > >> > >
>> > >> > > Previous discussion:
>> > >> > >
>> > >> > > [DISCUSS] Move Pulsar SQL to a separated repository?
>> > >> > > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> BR,
>> > >> Qiang Huang
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by tison <wa...@gmail.com>.
Thanks for your input!
I update the prototype as described in the proposal:
https://github.com/apache/pulsar/pull/17062
Will initialize a vote for PIP-200 soon.
Best,
tison.
Zhengxin Cai <ca...@gmail.com> 于2022年8月26日周五 08:36写道：
> +1 on this,
> I think helm chart should not be a big issue, we can just upgrade the chart
> and provide necessary doc on how to upgrade.
>
> tison <wa...@gmail.com> 于2022年8月24日周三 16:41写道：
>
> > Hi Enrico and Qiang,
> >
> > The issue description was updated. Picked here:
> >
> > > upgrade and downgrade doc
> >
> > This change should not affect those who use Pulsar with the entry point
> > script, but it changes the layout of the release artifact.
> >
> > I'm going to write a release note about this change and also post it on
> the
> > Pulsar SQL overview page as a caveat. Draft here:
> >
> > # Caveat
> >
> > If you're upgrading Pulsar SQL from 2.11 or early, you should copy the
> > related configs from `conf/presto` to `trino/conf`, and `lib/presto` to
> > `trino`. If you're downgrading Pulsar SQL to 2.11 or early from 2.12, do
> > verse visa.
> >
> > > Pulsar Helm Chart
> >
> > From what I understand, Pulsar Helm Chart is a wrapper of pulsar-all
> docker
> > image. I don't find any reference to presto/trino/sql effectively in the
> > codebase, so I think it's currently transparent for its users and the
> > caveat under Pulsar SQL overview page should be enough.
> >
> > Best,
> > tison.
> >
> >
> > tison <wa...@gmail.com> 于2022年8月17日周三 19:40写道：
> >
> > > Thanks for your feedback!
> > >
> > > 1. According to the upgrade and downgrade doc, I think the minimum
> > > requirements are a release note. Describe the layout change and how
> users
> > > should move the folder. I'll elaborate on the issue and notify you
> here.
> > > 2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how
> it
> > > can be relevant in days. If someone who maintains the Chart can provide
> > > some input, it will help!
> > > 3. "There are 3 issues". It's a description about the
> > > background/motivation, while we handle the first issue in this PIP.
> > > Although, subtasks can be divided into packaging changes, possible doc
> > > changes, and possible Chart changes.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Qiang Huang <qi...@gmail.com> 于2022年8月17日周三 19:25写道：
> > >
> > >> Looks good. I have two points:
> > >> 1. It is necessary to supplement the upgrade and downgrade
> documentation
> > >> in
> > >> Pulsar.
> > >> 2. There are 3 issues mentioned in the PIP, should we split it into 3
> > >> small
> > >> issues?
> > >>
> > >> Enrico Olivelli <eo...@gmail.com> 于2022年8月17日周三 17:30写道：
> > >>
> > >> > I generally agree with the PIP
> > >> >
> > >> > Can you please explain the interactions with the Pulsar Helm chart ?
> > >> > also we have to draw a migration path, because users that will
> upgrade
> > >> > Pulsar will have to move the configuration files in another location
> > >> >
> > >> > Enrico
> > >> >
> > >> > Il giorno mer 17 ago 2022 alle ore 11:15 tison <
> wander4096@gmail.com>
> > >> > ha scritto:
> > >> > >
> > >> > > Hello,
> > >> > >
> > >> > > This is a PIP to package the Pulsar Trino distro and config in a
> > >> > dedicated
> > >> > > folder.
> > >> > >
> > >> > > Link: https://github.com/apache/pulsar/issues/17137
> > >> > > Prototype: https://github.com/apache/pulsar/pull/17062
> > >> > >
> > >> > > Below you can find the proposal (I will amend the GH issue while
> we
> > >> > discuss
> > >> > > it).
> > >> > >
> > >> > > Best,
> > >> > > tison.
> > >> > >
> > >> > > Motivation
> > >> > > ========
> > >> > >
> > >> > > After https://github.com/apache/pulsar/pull/16683 merged, we
> > upgrade
> > >> > > PrestoSQL dependency in Pulsar SQL to the first several Trino
> > >> version. To
> > >> > > handle the name change cases and gradually refactor Pulsar SQL as
> a
> > >> > > self-contained module so that we can move it into a standalone
> > >> > repository,
> > >> > > I find that there're three major issues to resolve.
> > >> > >
> > >> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with
> > >> other
> > >> > > Pulsar configs.
> > >> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
> > >> > > 3. Integration tests of Pulsar SQL are tightly coupled with the
> main
> > >> repo
> > >> > > (test infra).
> > >> > >
> > >> > > This proposal is aimed at resolving the first issue to package
> > Pulsar
> > >> > Trino
> > >> > > distro and config in a dedicated folder; that is, to make it
> > >> > self-contained.
> > >> > >
> > >> > > Goal
> > >> > > ====
> > >> > >
> > >> > > I have already prepared a draft to perform the changes as
> > >> > > https://github.com/apache/pulsar/pull/17062. Generally, we move
> the
> > >> > config
> > >> > > files under `PRESTO_HOME` and correspondingly update scripts.
> > >> > >
> > >> > > In this way, all Trino distro artifacts are under the same home
> > path,
> > >> so
> > >> > > that we can later move it out as a whole.
> > >> > >
> > >> > > This change should not affect those who use Pulsar with the entry
> > >> point
> > >> > > script, but it changes the layout of the release artifact, so I'd
> > >> prefer
> > >> > to
> > >> > > perform a PIP process.
> > >> > >
> > >> > > Implementation
> > >> > > ============
> > >> > >
> > >> > > It's straightforward to inline in the "Goal" section.
> > >> > >
> > >> > > However, the name of the folder (`presto` or `trino`) and the
> level
> > of
> > >> > the
> > >> > > folder (`lib/presto/` or `trino/`) is open to discussion. I think
> > both
> > >> > are
> > >> > > fine and will try `trino/` first.
> > >> > >
> > >> > > To minimize unnecessary changes, I tend to keep the modules name
> > >> > > `pulsar-presto-xxx` as is.
> > >> > >
> > >> > > Alternatives
> > >> > > =========
> > >> > >
> > >> > > I don't make a completed proposal to resolve all three issues
> listed
> > >> > above.
> > >> > > Because I'm still unfamiliar with the latter two topics yet and
> I'd
> > >> > prefer
> > >> > > to implement these improvements one by one since they're naturally
> > >> > > independent. If I try to make a completed proposal at once, it's
> > >> highly
> > >> > > possible I give up halfway.
> > >> > >
> > >> > > Anything else?
> > >> > > ===========
> > >> > >
> > >> > > Previous discussion:
> > >> > >
> > >> > > [DISCUSS] Move Pulsar SQL to a separated repository?
> > >> > > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
> > >> >
> > >>
> > >>
> > >> --
> > >> BR,
> > >> Qiang Huang
> > >>
> > >
> >
>
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by Zhengxin Cai <ca...@gmail.com>.
+1 on this,
I think helm chart should not be a big issue, we can just upgrade the chart
and provide necessary doc on how to upgrade.
tison <wa...@gmail.com> 于2022年8月24日周三 16:41写道：
> Hi Enrico and Qiang,
>
> The issue description was updated. Picked here:
>
> > upgrade and downgrade doc
>
> This change should not affect those who use Pulsar with the entry point
> script, but it changes the layout of the release artifact.
>
> I'm going to write a release note about this change and also post it on the
> Pulsar SQL overview page as a caveat. Draft here:
>
> # Caveat
>
> If you're upgrading Pulsar SQL from 2.11 or early, you should copy the
> related configs from `conf/presto` to `trino/conf`, and `lib/presto` to
> `trino`. If you're downgrading Pulsar SQL to 2.11 or early from 2.12, do
> verse visa.
>
> > Pulsar Helm Chart
>
> From what I understand, Pulsar Helm Chart is a wrapper of pulsar-all docker
> image. I don't find any reference to presto/trino/sql effectively in the
> codebase, so I think it's currently transparent for its users and the
> caveat under Pulsar SQL overview page should be enough.
>
> Best,
> tison.
>
>
> tison <wa...@gmail.com> 于2022年8月17日周三 19:40写道：
>
> > Thanks for your feedback!
> >
> > 1. According to the upgrade and downgrade doc, I think the minimum
> > requirements are a release note. Describe the layout change and how users
> > should move the folder. I'll elaborate on the issue and notify you here.
> > 2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how it
> > can be relevant in days. If someone who maintains the Chart can provide
> > some input, it will help!
> > 3. "There are 3 issues". It's a description about the
> > background/motivation, while we handle the first issue in this PIP.
> > Although, subtasks can be divided into packaging changes, possible doc
> > changes, and possible Chart changes.
> >
> > Best,
> > tison.
> >
> >
> > Qiang Huang <qi...@gmail.com> 于2022年8月17日周三 19:25写道：
> >
> >> Looks good. I have two points:
> >> 1. It is necessary to supplement the upgrade and downgrade documentation
> >> in
> >> Pulsar.
> >> 2. There are 3 issues mentioned in the PIP, should we split it into 3
> >> small
> >> issues?
> >>
> >> Enrico Olivelli <eo...@gmail.com> 于2022年8月17日周三 17:30写道：
> >>
> >> > I generally agree with the PIP
> >> >
> >> > Can you please explain the interactions with the Pulsar Helm chart ?
> >> > also we have to draw a migration path, because users that will upgrade
> >> > Pulsar will have to move the configuration files in another location
> >> >
> >> > Enrico
> >> >
> >> > Il giorno mer 17 ago 2022 alle ore 11:15 tison <wa...@gmail.com>
> >> > ha scritto:
> >> > >
> >> > > Hello,
> >> > >
> >> > > This is a PIP to package the Pulsar Trino distro and config in a
> >> > dedicated
> >> > > folder.
> >> > >
> >> > > Link: https://github.com/apache/pulsar/issues/17137
> >> > > Prototype: https://github.com/apache/pulsar/pull/17062
> >> > >
> >> > > Below you can find the proposal (I will amend the GH issue while we
> >> > discuss
> >> > > it).
> >> > >
> >> > > Best,
> >> > > tison.
> >> > >
> >> > > Motivation
> >> > > ========
> >> > >
> >> > > After https://github.com/apache/pulsar/pull/16683 merged, we
> upgrade
> >> > > PrestoSQL dependency in Pulsar SQL to the first several Trino
> >> version. To
> >> > > handle the name change cases and gradually refactor Pulsar SQL as a
> >> > > self-contained module so that we can move it into a standalone
> >> > repository,
> >> > > I find that there're three major issues to resolve.
> >> > >
> >> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with
> >> other
> >> > > Pulsar configs.
> >> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
> >> > > 3. Integration tests of Pulsar SQL are tightly coupled with the main
> >> repo
> >> > > (test infra).
> >> > >
> >> > > This proposal is aimed at resolving the first issue to package
> Pulsar
> >> > Trino
> >> > > distro and config in a dedicated folder; that is, to make it
> >> > self-contained.
> >> > >
> >> > > Goal
> >> > > ====
> >> > >
> >> > > I have already prepared a draft to perform the changes as
> >> > > https://github.com/apache/pulsar/pull/17062. Generally, we move the
> >> > config
> >> > > files under `PRESTO_HOME` and correspondingly update scripts.
> >> > >
> >> > > In this way, all Trino distro artifacts are under the same home
> path,
> >> so
> >> > > that we can later move it out as a whole.
> >> > >
> >> > > This change should not affect those who use Pulsar with the entry
> >> point
> >> > > script, but it changes the layout of the release artifact, so I'd
> >> prefer
> >> > to
> >> > > perform a PIP process.
> >> > >
> >> > > Implementation
> >> > > ============
> >> > >
> >> > > It's straightforward to inline in the "Goal" section.
> >> > >
> >> > > However, the name of the folder (`presto` or `trino`) and the level
> of
> >> > the
> >> > > folder (`lib/presto/` or `trino/`) is open to discussion. I think
> both
> >> > are
> >> > > fine and will try `trino/` first.
> >> > >
> >> > > To minimize unnecessary changes, I tend to keep the modules name
> >> > > `pulsar-presto-xxx` as is.
> >> > >
> >> > > Alternatives
> >> > > =========
> >> > >
> >> > > I don't make a completed proposal to resolve all three issues listed
> >> > above.
> >> > > Because I'm still unfamiliar with the latter two topics yet and I'd
> >> > prefer
> >> > > to implement these improvements one by one since they're naturally
> >> > > independent. If I try to make a completed proposal at once, it's
> >> highly
> >> > > possible I give up halfway.
> >> > >
> >> > > Anything else?
> >> > > ===========
> >> > >
> >> > > Previous discussion:
> >> > >
> >> > > [DISCUSS] Move Pulsar SQL to a separated repository?
> >> > > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
> >> >
> >>
> >>
> >> --
> >> BR,
> >> Qiang Huang
> >>
> >
>
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by tison <wa...@gmail.com>.
Hi Enrico and Qiang,
The issue description was updated. Picked here:
> upgrade and downgrade doc
This change should not affect those who use Pulsar with the entry point
script, but it changes the layout of the release artifact.
I'm going to write a release note about this change and also post it on the
Pulsar SQL overview page as a caveat. Draft here:
# Caveat
If you're upgrading Pulsar SQL from 2.11 or early, you should copy the
related configs from `conf/presto` to `trino/conf`, and `lib/presto` to
`trino`. If you're downgrading Pulsar SQL to 2.11 or early from 2.12, do
verse visa.
> Pulsar Helm Chart
From what I understand, Pulsar Helm Chart is a wrapper of pulsar-all docker
image. I don't find any reference to presto/trino/sql effectively in the
codebase, so I think it's currently transparent for its users and the
caveat under Pulsar SQL overview page should be enough.
Best,
tison.
tison <wa...@gmail.com> 于2022年8月17日周三 19:40写道：
> Thanks for your feedback!
>
> 1. According to the upgrade and downgrade doc, I think the minimum
> requirements are a release note. Describe the layout change and how users
> should move the folder. I'll elaborate on the issue and notify you here.
> 2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how it
> can be relevant in days. If someone who maintains the Chart can provide
> some input, it will help!
> 3. "There are 3 issues". It's a description about the
> background/motivation, while we handle the first issue in this PIP.
> Although, subtasks can be divided into packaging changes, possible doc
> changes, and possible Chart changes.
>
> Best,
> tison.
>
>
> Qiang Huang <qi...@gmail.com> 于2022年8月17日周三 19:25写道：
>
>> Looks good. I have two points:
>> 1. It is necessary to supplement the upgrade and downgrade documentation
>> in
>> Pulsar.
>> 2. There are 3 issues mentioned in the PIP, should we split it into 3
>> small
>> issues?
>>
>> Enrico Olivelli <eo...@gmail.com> 于2022年8月17日周三 17:30写道：
>>
>> > I generally agree with the PIP
>> >
>> > Can you please explain the interactions with the Pulsar Helm chart ?
>> > also we have to draw a migration path, because users that will upgrade
>> > Pulsar will have to move the configuration files in another location
>> >
>> > Enrico
>> >
>> > Il giorno mer 17 ago 2022 alle ore 11:15 tison <wa...@gmail.com>
>> > ha scritto:
>> > >
>> > > Hello,
>> > >
>> > > This is a PIP to package the Pulsar Trino distro and config in a
>> > dedicated
>> > > folder.
>> > >
>> > > Link: https://github.com/apache/pulsar/issues/17137
>> > > Prototype: https://github.com/apache/pulsar/pull/17062
>> > >
>> > > Below you can find the proposal (I will amend the GH issue while we
>> > discuss
>> > > it).
>> > >
>> > > Best,
>> > > tison.
>> > >
>> > > Motivation
>> > > ========
>> > >
>> > > After https://github.com/apache/pulsar/pull/16683 merged, we upgrade
>> > > PrestoSQL dependency in Pulsar SQL to the first several Trino
>> version. To
>> > > handle the name change cases and gradually refactor Pulsar SQL as a
>> > > self-contained module so that we can move it into a standalone
>> > repository,
>> > > I find that there're three major issues to resolve.
>> > >
>> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with
>> other
>> > > Pulsar configs.
>> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
>> > > 3. Integration tests of Pulsar SQL are tightly coupled with the main
>> repo
>> > > (test infra).
>> > >
>> > > This proposal is aimed at resolving the first issue to package Pulsar
>> > Trino
>> > > distro and config in a dedicated folder; that is, to make it
>> > self-contained.
>> > >
>> > > Goal
>> > > ====
>> > >
>> > > I have already prepared a draft to perform the changes as
>> > > https://github.com/apache/pulsar/pull/17062. Generally, we move the
>> > config
>> > > files under `PRESTO_HOME` and correspondingly update scripts.
>> > >
>> > > In this way, all Trino distro artifacts are under the same home path,
>> so
>> > > that we can later move it out as a whole.
>> > >
>> > > This change should not affect those who use Pulsar with the entry
>> point
>> > > script, but it changes the layout of the release artifact, so I'd
>> prefer
>> > to
>> > > perform a PIP process.
>> > >
>> > > Implementation
>> > > ============
>> > >
>> > > It's straightforward to inline in the "Goal" section.
>> > >
>> > > However, the name of the folder (`presto` or `trino`) and the level of
>> > the
>> > > folder (`lib/presto/` or `trino/`) is open to discussion. I think both
>> > are
>> > > fine and will try `trino/` first.
>> > >
>> > > To minimize unnecessary changes, I tend to keep the modules name
>> > > `pulsar-presto-xxx` as is.
>> > >
>> > > Alternatives
>> > > =========
>> > >
>> > > I don't make a completed proposal to resolve all three issues listed
>> > above.
>> > > Because I'm still unfamiliar with the latter two topics yet and I'd
>> > prefer
>> > > to implement these improvements one by one since they're naturally
>> > > independent. If I try to make a completed proposal at once, it's
>> highly
>> > > possible I give up halfway.
>> > >
>> > > Anything else?
>> > > ===========
>> > >
>> > > Previous discussion:
>> > >
>> > > [DISCUSS] Move Pulsar SQL to a separated repository?
>> > > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
>> >
>>
>>
>> --
>> BR,
>> Qiang Huang
>>
>
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by tison <wa...@gmail.com>.
Thanks for your feedback!
1. According to the upgrade and downgrade doc, I think the minimum
requirements are a release note. Describe the layout change and how users
should move the folder. I'll elaborate on the issue and notify you here.
2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how it
can be relevant in days. If someone who maintains the Chart can provide
some input, it will help!
3. "There are 3 issues". It's a description about the
background/motivation, while we handle the first issue in this PIP.
Although, subtasks can be divided into packaging changes, possible doc
changes, and possible Chart changes.
Best,
tison.
Qiang Huang <qi...@gmail.com> 于2022年8月17日周三 19:25写道：
> Looks good. I have two points:
> 1. It is necessary to supplement the upgrade and downgrade documentation in
> Pulsar.
> 2. There are 3 issues mentioned in the PIP, should we split it into 3 small
> issues?
>
> Enrico Olivelli <eo...@gmail.com> 于2022年8月17日周三 17:30写道：
>
> > I generally agree with the PIP
> >
> > Can you please explain the interactions with the Pulsar Helm chart ?
> > also we have to draw a migration path, because users that will upgrade
> > Pulsar will have to move the configuration files in another location
> >
> > Enrico
> >
> > Il giorno mer 17 ago 2022 alle ore 11:15 tison <wa...@gmail.com>
> > ha scritto:
> > >
> > > Hello,
> > >
> > > This is a PIP to package the Pulsar Trino distro and config in a
> > dedicated
> > > folder.
> > >
> > > Link: https://github.com/apache/pulsar/issues/17137
> > > Prototype: https://github.com/apache/pulsar/pull/17062
> > >
> > > Below you can find the proposal (I will amend the GH issue while we
> > discuss
> > > it).
> > >
> > > Best,
> > > tison.
> > >
> > > Motivation
> > > ========
> > >
> > > After https://github.com/apache/pulsar/pull/16683 merged, we upgrade
> > > PrestoSQL dependency in Pulsar SQL to the first several Trino version.
> To
> > > handle the name change cases and gradually refactor Pulsar SQL as a
> > > self-contained module so that we can move it into a standalone
> > repository,
> > > I find that there're three major issues to resolve.
> > >
> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with other
> > > Pulsar configs.
> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
> > > 3. Integration tests of Pulsar SQL are tightly coupled with the main
> repo
> > > (test infra).
> > >
> > > This proposal is aimed at resolving the first issue to package Pulsar
> > Trino
> > > distro and config in a dedicated folder; that is, to make it
> > self-contained.
> > >
> > > Goal
> > > ====
> > >
> > > I have already prepared a draft to perform the changes as
> > > https://github.com/apache/pulsar/pull/17062. Generally, we move the
> > config
> > > files under `PRESTO_HOME` and correspondingly update scripts.
> > >
> > > In this way, all Trino distro artifacts are under the same home path,
> so
> > > that we can later move it out as a whole.
> > >
> > > This change should not affect those who use Pulsar with the entry point
> > > script, but it changes the layout of the release artifact, so I'd
> prefer
> > to
> > > perform a PIP process.
> > >
> > > Implementation
> > > ============
> > >
> > > It's straightforward to inline in the "Goal" section.
> > >
> > > However, the name of the folder (`presto` or `trino`) and the level of
> > the
> > > folder (`lib/presto/` or `trino/`) is open to discussion. I think both
> > are
> > > fine and will try `trino/` first.
> > >
> > > To minimize unnecessary changes, I tend to keep the modules name
> > > `pulsar-presto-xxx` as is.
> > >
> > > Alternatives
> > > =========
> > >
> > > I don't make a completed proposal to resolve all three issues listed
> > above.
> > > Because I'm still unfamiliar with the latter two topics yet and I'd
> > prefer
> > > to implement these improvements one by one since they're naturally
> > > independent. If I try to make a completed proposal at once, it's highly
> > > possible I give up halfway.
> > >
> > > Anything else?
> > > ===========
> > >
> > > Previous discussion:
> > >
> > > [DISCUSS] Move Pulsar SQL to a separated repository?
> > > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
> >
>
>
> --
> BR,
> Qiang Huang
>
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by Qiang Huang <qi...@gmail.com>.
Looks good. I have two points:
1. It is necessary to supplement the upgrade and downgrade documentation in
Pulsar.
2. There are 3 issues mentioned in the PIP, should we split it into 3 small
issues?
Enrico Olivelli <eo...@gmail.com> 于2022年8月17日周三 17:30写道：
> I generally agree with the PIP
>
> Can you please explain the interactions with the Pulsar Helm chart ?
> also we have to draw a migration path, because users that will upgrade
> Pulsar will have to move the configuration files in another location
>
> Enrico
>
> Il giorno mer 17 ago 2022 alle ore 11:15 tison <wa...@gmail.com>
> ha scritto:
> >
> > Hello,
> >
> > This is a PIP to package the Pulsar Trino distro and config in a
> dedicated
> > folder.
> >
> > Link: https://github.com/apache/pulsar/issues/17137
> > Prototype: https://github.com/apache/pulsar/pull/17062
> >
> > Below you can find the proposal (I will amend the GH issue while we
> discuss
> > it).
> >
> > Best,
> > tison.
> >
> > Motivation
> > ========
> >
> > After https://github.com/apache/pulsar/pull/16683 merged, we upgrade
> > PrestoSQL dependency in Pulsar SQL to the first several Trino version. To
> > handle the name change cases and gradually refactor Pulsar SQL as a
> > self-contained module so that we can move it into a standalone
> repository,
> > I find that there're three major issues to resolve.
> >
> > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with other
> > Pulsar configs.
> > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
> > 3. Integration tests of Pulsar SQL are tightly coupled with the main repo
> > (test infra).
> >
> > This proposal is aimed at resolving the first issue to package Pulsar
> Trino
> > distro and config in a dedicated folder; that is, to make it
> self-contained.
> >
> > Goal
> > ====
> >
> > I have already prepared a draft to perform the changes as
> > https://github.com/apache/pulsar/pull/17062. Generally, we move the
> config
> > files under `PRESTO_HOME` and correspondingly update scripts.
> >
> > In this way, all Trino distro artifacts are under the same home path, so
> > that we can later move it out as a whole.
> >
> > This change should not affect those who use Pulsar with the entry point
> > script, but it changes the layout of the release artifact, so I'd prefer
> to
> > perform a PIP process.
> >
> > Implementation
> > ============
> >
> > It's straightforward to inline in the "Goal" section.
> >
> > However, the name of the folder (`presto` or `trino`) and the level of
> the
> > folder (`lib/presto/` or `trino/`) is open to discussion. I think both
> are
> > fine and will try `trino/` first.
> >
> > To minimize unnecessary changes, I tend to keep the modules name
> > `pulsar-presto-xxx` as is.
> >
> > Alternatives
> > =========
> >
> > I don't make a completed proposal to resolve all three issues listed
> above.
> > Because I'm still unfamiliar with the latter two topics yet and I'd
> prefer
> > to implement these improvements one by one since they're naturally
> > independent. If I try to make a completed proposal at once, it's highly
> > possible I give up halfway.
> >
> > Anything else?
> > ===========
> >
> > Previous discussion:
> >
> > [DISCUSS] Move Pulsar SQL to a separated repository?
> > https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8
>
--
BR,
Qiang Huang
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Posted by Enrico Olivelli <eo...@gmail.com>.
I generally agree with the PIP
Can you please explain the interactions with the Pulsar Helm chart ?
also we have to draw a migration path, because users that will upgrade
Pulsar will have to move the configuration files in another location
Enrico
Il giorno mer 17 ago 2022 alle ore 11:15 tison <wa...@gmail.com>
ha scritto:
>
> Hello,
>
> This is a PIP to package the Pulsar Trino distro and config in a dedicated
> folder.
>
> Link: https://github.com/apache/pulsar/issues/17137
> Prototype: https://github.com/apache/pulsar/pull/17062
>
> Below you can find the proposal (I will amend the GH issue while we discuss
> it).
>
> Best,
> tison.
>
> Motivation
> ========
>
> After https://github.com/apache/pulsar/pull/16683 merged, we upgrade
> PrestoSQL dependency in Pulsar SQL to the first several Trino version. To
> handle the name change cases and gradually refactor Pulsar SQL as a
> self-contained module so that we can move it into a standalone repository,
> I find that there're three major issues to resolve.
>
> 1. Configs of Pulsar SQL go under the `conf/` folder and mix with other
> Pulsar configs.
> 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
> 3. Integration tests of Pulsar SQL are tightly coupled with the main repo
> (test infra).
>
> This proposal is aimed at resolving the first issue to package Pulsar Trino
> distro and config in a dedicated folder; that is, to make it self-contained.
>
> Goal
> ====
>
> I have already prepared a draft to perform the changes as
> https://github.com/apache/pulsar/pull/17062. Generally, we move the config
> files under `PRESTO_HOME` and correspondingly update scripts.
>
> In this way, all Trino distro artifacts are under the same home path, so
> that we can later move it out as a whole.
>
> This change should not affect those who use Pulsar with the entry point
> script, but it changes the layout of the release artifact, so I'd prefer to
> perform a PIP process.
>
> Implementation
> ============
>
> It's straightforward to inline in the "Goal" section.
>
> However, the name of the folder (`presto` or `trino`) and the level of the
> folder (`lib/presto/` or `trino/`) is open to discussion. I think both are
> fine and will try `trino/` first.
>
> To minimize unnecessary changes, I tend to keep the modules name
> `pulsar-presto-xxx` as is.
>
> Alternatives
> =========
>
> I don't make a completed proposal to resolve all three issues listed above.
> Because I'm still unfamiliar with the latter two topics yet and I'd prefer
> to implement these improvements one by one since they're naturally
> independent. If I try to make a completed proposal at once, it's highly
> possible I give up halfway.
>
> Anything else?
> ===========
>
> Previous discussion:
>
> [DISCUSS] Move Pulsar SQL to a separated repository?
> https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8