You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Dmitry Sysolyatin <dm...@gmail.com> on 2022/09/10 15:29:48 UTC
Does someone interested in integration calcite + PostgreSQL wire protocol ?
Hello!
I've been working on the integration calcite and PostgreSQL wire protocol
[1] to emulate Postgres server (Of course it includes emulating
`pg_catalog` schema). It gives a user the ability to connect to a calcite
schema using PostgreSQL tools and libraries.
Our team is thinking to open source code of this project. And at first, we
would like to know if anyone would be interested before doing any next
steps.
If you would be interested in, just write +1
[1] https://www.postgresql.org/docs/current/protocol.html
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by JiaTao Tao <ta...@gmail.com>.
+1
Regards!
Aron Tao
Julian Hyde <jh...@gmail.com> 于2022年10月2日周日 05:06写道：
> Thanks for trying, Dmitry. This happens a lot in commercial open source
> (see Calcite’s Spark adapter, for instance) but once in a while you get
> lucky, and that makes it worthwhile.
>
> Julian
>
>
> > On Sep 28, 2022, at 3:18 AM, Dmitry Sysolyatin <dm...@gmail.com>
> wrote:
> >
> > Hello!
> > I had a long discussion with our company management where I tried to
> > convince them to publish those code as open source. But unfortunately the
> > answer was negative.
> >
> > Sorry
> >
> >
> > On Mon, Sep 12, 2022 at 8:22 AM André kouamé <ko...@gmail.com>
> > wrote:
> >
> >> +1
> >>
> >> Le lun. 12 sept. 2022 à 00:36, Justin Swanhart <gr...@gmail.com> a
> >> écrit :
> >>
> >>> +1
> >>>
> >>> On Sun, Sep 11, 2022 at 10:50 AM Gavin Ray <ra...@gmail.com>
> >> wrote:
> >>>
> >>>> Understandable, no worries =)
> >>>>
> >>>> On Sun, Sep 11, 2022 at 2:49 AM Dmitry Sysolyatin <
> >>> dm.sysolyatin@gmail.com
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> unfortunately, I can't reveal the details of the implementation until
> >>> our
> >>>>> company has given the official right to make this project open source
> >>> :(
> >>>>>
> >>>>>> 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com>
> >>>> написал(а):
> >>>>>>
> >>>>>> Did you wind up using CrateDB's PG implementation?
> >>>>>> I found it was too tied to CrateDB specific stuff, not really
> >> usable
> >>>>>> without trying to rewrite a lot of it
> >>>>>>
> >>>>>> So instead I wrote a framework where you can implement an interface
> >>> for
> >>>>>> handling each
> >>>>>> PG wire message type and it does the rest (encoding/decoding,
> >>>> networking,
> >>>>>> etc) using Netty
> >>>>>>
> >>>>>> (I published it here, it's in Kotlin but I also have a Java
> >> version I
> >>>>> mean
> >>>>>> to publish this weekend)
> >>>>>> GavinRay97/PgProtoKt: A framework for building Postgres
> >> Wire-Protocol
> >>>>>> backends on the JVM, backed by Netty (github.com)
> >>>>>> <https://github.com/GavinRay97/PgProtoKt#example>
> >>>>>>
> >>>>>>
> >>>>>>> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <
> >>>>> dm.sysolyatin@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> Our first implementation is also used external PG. But there were
> >> 3
> >>>>>>> problem with that:
> >>>>>>> 1. Need external pg setup like you said
> >>>>>>> 2. Need to synchronise external pg with list of your tables and
> >>>> columns
> >>>>>>> 3. Security, it is hard to do any security related things on top
> >> of
> >>>>>>> external Postgres pg_catalog table. For example, to show user only
> >>> his
> >>>>>>> tables, schemas and so on, so on
> >>>>>>>
> >>>>>>>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com>
> >>>>> написал(а):
> >>>>>>>>
> >>>>>>>> I have a basic JVM framework for building PG wire protocol
> >>> services
> >>>>>>> that is
> >>>>>>>> integrated with Calcite
> >>>>>>>> But it does not support pg_catalog, more just passes queries
> >>> directly
> >>>>> to
> >>>>>>>> Calcite
> >>>>>>>>
> >>>>>>>> A hacky way to add support for pg_catalog is to use pg_fdw with a
> >>>> real
> >>>>>>>> Postgres DB so that it takes care of the metadata
> >>>>>>>> But this is not ideal and it requires having an external Postgres
> >>>> data
> >>>>> +
> >>>>>>>> some setup
> >>>>>>>>
> >>>>>>>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <
> >> ray.gavin97@gmail.com
> >>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> +1
> >>>>>>>>>
> >>>>>>>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> >>>>>>>>> dm.sysolyatin@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hello!
> >>>>>>>>>>
> >>>>>>>>>> I've been working on the integration calcite and PostgreSQL
> >> wire
> >>>>>>> protocol
> >>>>>>>>>> [1] to emulate Postgres server (Of course it includes emulating
> >>>>>>>>>> `pg_catalog` schema). It gives a user the ability to connect
> >> to a
> >>>>>>> calcite
> >>>>>>>>>> schema using PostgreSQL tools and libraries.
> >>>>>>>>>>
> >>>>>>>>>> Our team is thinking to open source code of this project. And
> >> at
> >>>>>>> first, we
> >>>>>>>>>> would like to know if anyone would be interested before doing
> >> any
> >>>>> next
> >>>>>>>>>> steps.
> >>>>>>>>>>
> >>>>>>>>>> If you would be interested in, just write +1
> >>>>>>>>>>
> >>>>>>>>>> [1] https://www.postgresql.org/docs/current/protocol.html
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Julian Hyde <jh...@gmail.com>.
Thanks for trying, Dmitry. This happens a lot in commercial open source (see Calcite’s Spark adapter, for instance) but once in a while you get lucky, and that makes it worthwhile.
Julian
> On Sep 28, 2022, at 3:18 AM, Dmitry Sysolyatin <dm...@gmail.com> wrote:
>
> Hello!
> I had a long discussion with our company management where I tried to
> convince them to publish those code as open source. But unfortunately the
> answer was negative.
>
> Sorry
>
>
> On Mon, Sep 12, 2022 at 8:22 AM André kouamé <ko...@gmail.com>
> wrote:
>
>> +1
>>
>> Le lun. 12 sept. 2022 à 00:36, Justin Swanhart <gr...@gmail.com> a
>> écrit :
>>
>>> +1
>>>
>>> On Sun, Sep 11, 2022 at 10:50 AM Gavin Ray <ra...@gmail.com>
>> wrote:
>>>
>>>> Understandable, no worries =)
>>>>
>>>> On Sun, Sep 11, 2022 at 2:49 AM Dmitry Sysolyatin <
>>> dm.sysolyatin@gmail.com
>>>>>
>>>> wrote:
>>>>
>>>>> unfortunately, I can't reveal the details of the implementation until
>>> our
>>>>> company has given the official right to make this project open source
>>> :(
>>>>>
>>>>>> 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com>
>>>> написал(а):
>>>>>>
>>>>>> Did you wind up using CrateDB's PG implementation?
>>>>>> I found it was too tied to CrateDB specific stuff, not really
>> usable
>>>>>> without trying to rewrite a lot of it
>>>>>>
>>>>>> So instead I wrote a framework where you can implement an interface
>>> for
>>>>>> handling each
>>>>>> PG wire message type and it does the rest (encoding/decoding,
>>>> networking,
>>>>>> etc) using Netty
>>>>>>
>>>>>> (I published it here, it's in Kotlin but I also have a Java
>> version I
>>>>> mean
>>>>>> to publish this weekend)
>>>>>> GavinRay97/PgProtoKt: A framework for building Postgres
>> Wire-Protocol
>>>>>> backends on the JVM, backed by Netty (github.com)
>>>>>> <https://github.com/GavinRay97/PgProtoKt#example>
>>>>>>
>>>>>>
>>>>>>> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <
>>>>> dm.sysolyatin@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Our first implementation is also used external PG. But there were
>> 3
>>>>>>> problem with that:
>>>>>>> 1. Need external pg setup like you said
>>>>>>> 2. Need to synchronise external pg with list of your tables and
>>>> columns
>>>>>>> 3. Security, it is hard to do any security related things on top
>> of
>>>>>>> external Postgres pg_catalog table. For example, to show user only
>>> his
>>>>>>> tables, schemas and so on, so on
>>>>>>>
>>>>>>>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com>
>>>>> написал(а):
>>>>>>>>
>>>>>>>> I have a basic JVM framework for building PG wire protocol
>>> services
>>>>>>> that is
>>>>>>>> integrated with Calcite
>>>>>>>> But it does not support pg_catalog, more just passes queries
>>> directly
>>>>> to
>>>>>>>> Calcite
>>>>>>>>
>>>>>>>> A hacky way to add support for pg_catalog is to use pg_fdw with a
>>>> real
>>>>>>>> Postgres DB so that it takes care of the metadata
>>>>>>>> But this is not ideal and it requires having an external Postgres
>>>> data
>>>>> +
>>>>>>>> some setup
>>>>>>>>
>>>>>>>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <
>> ray.gavin97@gmail.com
>>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
>>>>>>>>> dm.sysolyatin@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello!
>>>>>>>>>>
>>>>>>>>>> I've been working on the integration calcite and PostgreSQL
>> wire
>>>>>>> protocol
>>>>>>>>>> [1] to emulate Postgres server (Of course it includes emulating
>>>>>>>>>> `pg_catalog` schema). It gives a user the ability to connect
>> to a
>>>>>>> calcite
>>>>>>>>>> schema using PostgreSQL tools and libraries.
>>>>>>>>>>
>>>>>>>>>> Our team is thinking to open source code of this project. And
>> at
>>>>>>> first, we
>>>>>>>>>> would like to know if anyone would be interested before doing
>> any
>>>>> next
>>>>>>>>>> steps.
>>>>>>>>>>
>>>>>>>>>> If you would be interested in, just write +1
>>>>>>>>>>
>>>>>>>>>> [1] https://www.postgresql.org/docs/current/protocol.html
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Dmitry Sysolyatin <dm...@gmail.com>.
Hello!
I had a long discussion with our company management where I tried to
convince them to publish those code as open source. But unfortunately the
answer was negative.
Sorry
On Mon, Sep 12, 2022 at 8:22 AM André kouamé <ko...@gmail.com>
wrote:
> +1
>
> Le lun. 12 sept. 2022 à 00:36, Justin Swanhart <gr...@gmail.com> a
> écrit :
>
> > +1
> >
> > On Sun, Sep 11, 2022 at 10:50 AM Gavin Ray <ra...@gmail.com>
> wrote:
> >
> > > Understandable, no worries =)
> > >
> > > On Sun, Sep 11, 2022 at 2:49 AM Dmitry Sysolyatin <
> > dm.sysolyatin@gmail.com
> > > >
> > > wrote:
> > >
> > > > unfortunately, I can't reveal the details of the implementation until
> > our
> > > > company has given the official right to make this project open source
> > :(
> > > >
> > > > > 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com>
> > > написал(а):
> > > > >
> > > > > Did you wind up using CrateDB's PG implementation?
> > > > > I found it was too tied to CrateDB specific stuff, not really
> usable
> > > > > without trying to rewrite a lot of it
> > > > >
> > > > > So instead I wrote a framework where you can implement an interface
> > for
> > > > > handling each
> > > > > PG wire message type and it does the rest (encoding/decoding,
> > > networking,
> > > > > etc) using Netty
> > > > >
> > > > > (I published it here, it's in Kotlin but I also have a Java
> version I
> > > > mean
> > > > > to publish this weekend)
> > > > > GavinRay97/PgProtoKt: A framework for building Postgres
> Wire-Protocol
> > > > > backends on the JVM, backed by Netty (github.com)
> > > > > <https://github.com/GavinRay97/PgProtoKt#example>
> > > > >
> > > > >
> > > > >> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <
> > > > dm.sysolyatin@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>
> > > > >> Our first implementation is also used external PG. But there were
> 3
> > > > >> problem with that:
> > > > >> 1. Need external pg setup like you said
> > > > >> 2. Need to synchronise external pg with list of your tables and
> > > columns
> > > > >> 3. Security, it is hard to do any security related things on top
> of
> > > > >> external Postgres pg_catalog table. For example, to show user only
> > his
> > > > >> tables, schemas and so on, so on
> > > > >>
> > > > >>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com>
> > > > написал(а):
> > > > >>>
> > > > >>> I have a basic JVM framework for building PG wire protocol
> > services
> > > > >> that is
> > > > >>> integrated with Calcite
> > > > >>> But it does not support pg_catalog, more just passes queries
> > directly
> > > > to
> > > > >>> Calcite
> > > > >>>
> > > > >>> A hacky way to add support for pg_catalog is to use pg_fdw with a
> > > real
> > > > >>> Postgres DB so that it takes care of the metadata
> > > > >>> But this is not ideal and it requires having an external Postgres
> > > data
> > > > +
> > > > >>> some setup
> > > > >>>
> > > > >>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <
> ray.gavin97@gmail.com
> > >
> > > > >> wrote:
> > > > >>>>
> > > > >>>> +1
> > > > >>>>
> > > > >>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> > > > >>>> dm.sysolyatin@gmail.com> wrote:
> > > > >>>>
> > > > >>>>> Hello!
> > > > >>>>>
> > > > >>>>> I've been working on the integration calcite and PostgreSQL
> wire
> > > > >> protocol
> > > > >>>>> [1] to emulate Postgres server (Of course it includes emulating
> > > > >>>>> `pg_catalog` schema). It gives a user the ability to connect
> to a
> > > > >> calcite
> > > > >>>>> schema using PostgreSQL tools and libraries.
> > > > >>>>>
> > > > >>>>> Our team is thinking to open source code of this project. And
> at
> > > > >> first, we
> > > > >>>>> would like to know if anyone would be interested before doing
> any
> > > > next
> > > > >>>>> steps.
> > > > >>>>>
> > > > >>>>> If you would be interested in, just write +1
> > > > >>>>>
> > > > >>>>> [1] https://www.postgresql.org/docs/current/protocol.html
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by André kouamé <ko...@gmail.com>.
+1
Le lun. 12 sept. 2022 à 00:36, Justin Swanhart <gr...@gmail.com> a
écrit :
> +1
>
> On Sun, Sep 11, 2022 at 10:50 AM Gavin Ray <ra...@gmail.com> wrote:
>
> > Understandable, no worries =)
> >
> > On Sun, Sep 11, 2022 at 2:49 AM Dmitry Sysolyatin <
> dm.sysolyatin@gmail.com
> > >
> > wrote:
> >
> > > unfortunately, I can't reveal the details of the implementation until
> our
> > > company has given the official right to make this project open source
> :(
> > >
> > > > 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com>
> > написал(а):
> > > >
> > > > Did you wind up using CrateDB's PG implementation?
> > > > I found it was too tied to CrateDB specific stuff, not really usable
> > > > without trying to rewrite a lot of it
> > > >
> > > > So instead I wrote a framework where you can implement an interface
> for
> > > > handling each
> > > > PG wire message type and it does the rest (encoding/decoding,
> > networking,
> > > > etc) using Netty
> > > >
> > > > (I published it here, it's in Kotlin but I also have a Java version I
> > > mean
> > > > to publish this weekend)
> > > > GavinRay97/PgProtoKt: A framework for building Postgres Wire-Protocol
> > > > backends on the JVM, backed by Netty (github.com)
> > > > <https://github.com/GavinRay97/PgProtoKt#example>
> > > >
> > > >
> > > >> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <
> > > dm.sysolyatin@gmail.com>
> > > >> wrote:
> > > >>
> > > >>
> > > >> Our first implementation is also used external PG. But there were 3
> > > >> problem with that:
> > > >> 1. Need external pg setup like you said
> > > >> 2. Need to synchronise external pg with list of your tables and
> > columns
> > > >> 3. Security, it is hard to do any security related things on top of
> > > >> external Postgres pg_catalog table. For example, to show user only
> his
> > > >> tables, schemas and so on, so on
> > > >>
> > > >>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com>
> > > написал(а):
> > > >>>
> > > >>> I have a basic JVM framework for building PG wire protocol
> services
> > > >> that is
> > > >>> integrated with Calcite
> > > >>> But it does not support pg_catalog, more just passes queries
> directly
> > > to
> > > >>> Calcite
> > > >>>
> > > >>> A hacky way to add support for pg_catalog is to use pg_fdw with a
> > real
> > > >>> Postgres DB so that it takes care of the metadata
> > > >>> But this is not ideal and it requires having an external Postgres
> > data
> > > +
> > > >>> some setup
> > > >>>
> > > >>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ray.gavin97@gmail.com
> >
> > > >> wrote:
> > > >>>>
> > > >>>> +1
> > > >>>>
> > > >>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> > > >>>> dm.sysolyatin@gmail.com> wrote:
> > > >>>>
> > > >>>>> Hello!
> > > >>>>>
> > > >>>>> I've been working on the integration calcite and PostgreSQL wire
> > > >> protocol
> > > >>>>> [1] to emulate Postgres server (Of course it includes emulating
> > > >>>>> `pg_catalog` schema). It gives a user the ability to connect to a
> > > >> calcite
> > > >>>>> schema using PostgreSQL tools and libraries.
> > > >>>>>
> > > >>>>> Our team is thinking to open source code of this project. And at
> > > >> first, we
> > > >>>>> would like to know if anyone would be interested before doing any
> > > next
> > > >>>>> steps.
> > > >>>>>
> > > >>>>> If you would be interested in, just write +1
> > > >>>>>
> > > >>>>> [1] https://www.postgresql.org/docs/current/protocol.html
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Justin Swanhart <gr...@gmail.com>.
+1
On Sun, Sep 11, 2022 at 10:50 AM Gavin Ray <ra...@gmail.com> wrote:
> Understandable, no worries =)
>
> On Sun, Sep 11, 2022 at 2:49 AM Dmitry Sysolyatin <dm.sysolyatin@gmail.com
> >
> wrote:
>
> > unfortunately, I can't reveal the details of the implementation until our
> > company has given the official right to make this project open source :(
> >
> > > 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com>
> написал(а):
> > >
> > > Did you wind up using CrateDB's PG implementation?
> > > I found it was too tied to CrateDB specific stuff, not really usable
> > > without trying to rewrite a lot of it
> > >
> > > So instead I wrote a framework where you can implement an interface for
> > > handling each
> > > PG wire message type and it does the rest (encoding/decoding,
> networking,
> > > etc) using Netty
> > >
> > > (I published it here, it's in Kotlin but I also have a Java version I
> > mean
> > > to publish this weekend)
> > > GavinRay97/PgProtoKt: A framework for building Postgres Wire-Protocol
> > > backends on the JVM, backed by Netty (github.com)
> > > <https://github.com/GavinRay97/PgProtoKt#example>
> > >
> > >
> > >> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <
> > dm.sysolyatin@gmail.com>
> > >> wrote:
> > >>
> > >>
> > >> Our first implementation is also used external PG. But there were 3
> > >> problem with that:
> > >> 1. Need external pg setup like you said
> > >> 2. Need to synchronise external pg with list of your tables and
> columns
> > >> 3. Security, it is hard to do any security related things on top of
> > >> external Postgres pg_catalog table. For example, to show user only his
> > >> tables, schemas and so on, so on
> > >>
> > >>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com>
> > написал(а):
> > >>>
> > >>> I have a basic JVM framework for building PG wire protocol services
> > >> that is
> > >>> integrated with Calcite
> > >>> But it does not support pg_catalog, more just passes queries directly
> > to
> > >>> Calcite
> > >>>
> > >>> A hacky way to add support for pg_catalog is to use pg_fdw with a
> real
> > >>> Postgres DB so that it takes care of the metadata
> > >>> But this is not ideal and it requires having an external Postgres
> data
> > +
> > >>> some setup
> > >>>
> > >>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ra...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>> +1
> > >>>>
> > >>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> > >>>> dm.sysolyatin@gmail.com> wrote:
> > >>>>
> > >>>>> Hello!
> > >>>>>
> > >>>>> I've been working on the integration calcite and PostgreSQL wire
> > >> protocol
> > >>>>> [1] to emulate Postgres server (Of course it includes emulating
> > >>>>> `pg_catalog` schema). It gives a user the ability to connect to a
> > >> calcite
> > >>>>> schema using PostgreSQL tools and libraries.
> > >>>>>
> > >>>>> Our team is thinking to open source code of this project. And at
> > >> first, we
> > >>>>> would like to know if anyone would be interested before doing any
> > next
> > >>>>> steps.
> > >>>>>
> > >>>>> If you would be interested in, just write +1
> > >>>>>
> > >>>>> [1] https://www.postgresql.org/docs/current/protocol.html
> > >>>>>
> > >>>>
> > >>
> >
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Gavin Ray <ra...@gmail.com>.
Understandable, no worries =)
On Sun, Sep 11, 2022 at 2:49 AM Dmitry Sysolyatin <dm...@gmail.com>
wrote:
> unfortunately, I can't reveal the details of the implementation until our
> company has given the official right to make this project open source :(
>
> > 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com> написал(а):
> >
> > Did you wind up using CrateDB's PG implementation?
> > I found it was too tied to CrateDB specific stuff, not really usable
> > without trying to rewrite a lot of it
> >
> > So instead I wrote a framework where you can implement an interface for
> > handling each
> > PG wire message type and it does the rest (encoding/decoding, networking,
> > etc) using Netty
> >
> > (I published it here, it's in Kotlin but I also have a Java version I
> mean
> > to publish this weekend)
> > GavinRay97/PgProtoKt: A framework for building Postgres Wire-Protocol
> > backends on the JVM, backed by Netty (github.com)
> > <https://github.com/GavinRay97/PgProtoKt#example>
> >
> >
> >> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <
> dm.sysolyatin@gmail.com>
> >> wrote:
> >>
> >>
> >> Our first implementation is also used external PG. But there were 3
> >> problem with that:
> >> 1. Need external pg setup like you said
> >> 2. Need to synchronise external pg with list of your tables and columns
> >> 3. Security, it is hard to do any security related things on top of
> >> external Postgres pg_catalog table. For example, to show user only his
> >> tables, schemas and so on, so on
> >>
> >>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com>
> написал(а):
> >>>
> >>> I have a basic JVM framework for building PG wire protocol services
> >> that is
> >>> integrated with Calcite
> >>> But it does not support pg_catalog, more just passes queries directly
> to
> >>> Calcite
> >>>
> >>> A hacky way to add support for pg_catalog is to use pg_fdw with a real
> >>> Postgres DB so that it takes care of the metadata
> >>> But this is not ideal and it requires having an external Postgres data
> +
> >>> some setup
> >>>
> >>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ra...@gmail.com>
> >> wrote:
> >>>>
> >>>> +1
> >>>>
> >>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> >>>> dm.sysolyatin@gmail.com> wrote:
> >>>>
> >>>>> Hello!
> >>>>>
> >>>>> I've been working on the integration calcite and PostgreSQL wire
> >> protocol
> >>>>> [1] to emulate Postgres server (Of course it includes emulating
> >>>>> `pg_catalog` schema). It gives a user the ability to connect to a
> >> calcite
> >>>>> schema using PostgreSQL tools and libraries.
> >>>>>
> >>>>> Our team is thinking to open source code of this project. And at
> >> first, we
> >>>>> would like to know if anyone would be interested before doing any
> next
> >>>>> steps.
> >>>>>
> >>>>> If you would be interested in, just write +1
> >>>>>
> >>>>> [1] https://www.postgresql.org/docs/current/protocol.html
> >>>>>
> >>>>
> >>
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Dmitry Sysolyatin <dm...@gmail.com>.
unfortunately, I can't reveal the details of the implementation until our company has given the official right to make this project open source :(
> 10 сент. 2022 г., в 20:16, Gavin Ray <ra...@gmail.com> написал(а):
>
> Did you wind up using CrateDB's PG implementation?
> I found it was too tied to CrateDB specific stuff, not really usable
> without trying to rewrite a lot of it
>
> So instead I wrote a framework where you can implement an interface for
> handling each
> PG wire message type and it does the rest (encoding/decoding, networking,
> etc) using Netty
>
> (I published it here, it's in Kotlin but I also have a Java version I mean
> to publish this weekend)
> GavinRay97/PgProtoKt: A framework for building Postgres Wire-Protocol
> backends on the JVM, backed by Netty (github.com)
> <https://github.com/GavinRay97/PgProtoKt#example>
>
>
>> On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <dm...@gmail.com>
>> wrote:
>>
>>
>> Our first implementation is also used external PG. But there were 3
>> problem with that:
>> 1. Need external pg setup like you said
>> 2. Need to synchronise external pg with list of your tables and columns
>> 3. Security, it is hard to do any security related things on top of
>> external Postgres pg_catalog table. For example, to show user only his
>> tables, schemas and so on, so on
>>
>>>> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com> написал(а):
>>>
>>> I have a basic JVM framework for building PG wire protocol services
>> that is
>>> integrated with Calcite
>>> But it does not support pg_catalog, more just passes queries directly to
>>> Calcite
>>>
>>> A hacky way to add support for pg_catalog is to use pg_fdw with a real
>>> Postgres DB so that it takes care of the metadata
>>> But this is not ideal and it requires having an external Postgres data +
>>> some setup
>>>
>>>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ra...@gmail.com>
>> wrote:
>>>>
>>>> +1
>>>>
>>>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
>>>> dm.sysolyatin@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I've been working on the integration calcite and PostgreSQL wire
>> protocol
>>>>> [1] to emulate Postgres server (Of course it includes emulating
>>>>> `pg_catalog` schema). It gives a user the ability to connect to a
>> calcite
>>>>> schema using PostgreSQL tools and libraries.
>>>>>
>>>>> Our team is thinking to open source code of this project. And at
>> first, we
>>>>> would like to know if anyone would be interested before doing any next
>>>>> steps.
>>>>>
>>>>> If you would be interested in, just write +1
>>>>>
>>>>> [1] https://www.postgresql.org/docs/current/protocol.html
>>>>>
>>>>
>>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Gavin Ray <ra...@gmail.com>.
Did you wind up using CrateDB's PG implementation?
I found it was too tied to CrateDB specific stuff, not really usable
without trying to rewrite a lot of it
So instead I wrote a framework where you can implement an interface for
handling each
PG wire message type and it does the rest (encoding/decoding, networking,
etc) using Netty
(I published it here, it's in Kotlin but I also have a Java version I mean
to publish this weekend)
GavinRay97/PgProtoKt: A framework for building Postgres Wire-Protocol
backends on the JVM, backed by Netty (github.com)
<https://github.com/GavinRay97/PgProtoKt#example>
On Sat, Sep 10, 2022 at 12:10 PM Dmitry Sysolyatin <dm...@gmail.com>
wrote:
>
> Our first implementation is also used external PG. But there were 3
> problem with that:
> 1. Need external pg setup like you said
> 2. Need to synchronise external pg with list of your tables and columns
> 3. Security, it is hard to do any security related things on top of
> external Postgres pg_catalog table. For example, to show user only his
> tables, schemas and so on, so on
>
> > 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com> написал(а):
> >
> > I have a basic JVM framework for building PG wire protocol services
> that is
> > integrated with Calcite
> > But it does not support pg_catalog, more just passes queries directly to
> > Calcite
> >
> > A hacky way to add support for pg_catalog is to use pg_fdw with a real
> > Postgres DB so that it takes care of the metadata
> > But this is not ideal and it requires having an external Postgres data +
> > some setup
> >
> >> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ra...@gmail.com>
> wrote:
> >>
> >> +1
> >>
> >> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> >> dm.sysolyatin@gmail.com> wrote:
> >>
> >>> Hello!
> >>>
> >>> I've been working on the integration calcite and PostgreSQL wire
> protocol
> >>> [1] to emulate Postgres server (Of course it includes emulating
> >>> `pg_catalog` schema). It gives a user the ability to connect to a
> calcite
> >>> schema using PostgreSQL tools and libraries.
> >>>
> >>> Our team is thinking to open source code of this project. And at
> first, we
> >>> would like to know if anyone would be interested before doing any next
> >>> steps.
> >>>
> >>> If you would be interested in, just write +1
> >>>
> >>> [1] https://www.postgresql.org/docs/current/protocol.html
> >>>
> >>
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Dmitry Sysolyatin <dm...@gmail.com>.
Our first implementation is also used external PG. But there were 3 problem with that:
1. Need external pg setup like you said
2. Need to synchronise external pg with list of your tables and columns
3. Security, it is hard to do any security related things on top of external Postgres pg_catalog table. For example, to show user only his tables, schemas and so on, so on
> 10 сент. 2022 г., в 18:58, Gavin Ray <ra...@gmail.com> написал(а):
>
> I have a basic JVM framework for building PG wire protocol services that is
> integrated with Calcite
> But it does not support pg_catalog, more just passes queries directly to
> Calcite
>
> A hacky way to add support for pg_catalog is to use pg_fdw with a real
> Postgres DB so that it takes care of the metadata
> But this is not ideal and it requires having an external Postgres data +
> some setup
>
>> On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ra...@gmail.com> wrote:
>>
>> +1
>>
>> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
>> dm.sysolyatin@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I've been working on the integration calcite and PostgreSQL wire protocol
>>> [1] to emulate Postgres server (Of course it includes emulating
>>> `pg_catalog` schema). It gives a user the ability to connect to a calcite
>>> schema using PostgreSQL tools and libraries.
>>>
>>> Our team is thinking to open source code of this project. And at first, we
>>> would like to know if anyone would be interested before doing any next
>>> steps.
>>>
>>> If you would be interested in, just write +1
>>>
>>> [1] https://www.postgresql.org/docs/current/protocol.html
>>>
>>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Gavin Ray <ra...@gmail.com>.
I have a basic JVM framework for building PG wire protocol services that is
integrated with Calcite
But it does not support pg_catalog, more just passes queries directly to
Calcite
A hacky way to add support for pg_catalog is to use pg_fdw with a real
Postgres DB so that it takes care of the metadata
But this is not ideal and it requires having an external Postgres data +
some setup
On Sat, Sep 10, 2022 at 11:55 AM Gavin Ray <ra...@gmail.com> wrote:
> +1
>
> On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <
> dm.sysolyatin@gmail.com> wrote:
>
>> Hello!
>>
>> I've been working on the integration calcite and PostgreSQL wire protocol
>> [1] to emulate Postgres server (Of course it includes emulating
>> `pg_catalog` schema). It gives a user the ability to connect to a calcite
>> schema using PostgreSQL tools and libraries.
>>
>> Our team is thinking to open source code of this project. And at first, we
>> would like to know if anyone would be interested before doing any next
>> steps.
>>
>> If you would be interested in, just write +1
>>
>> [1] https://www.postgresql.org/docs/current/protocol.html
>>
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Gavin Ray <ra...@gmail.com>.
+1
On Sat, Sep 10, 2022 at 11:30 AM Dmitry Sysolyatin <dm...@gmail.com>
wrote:
> Hello!
>
> I've been working on the integration calcite and PostgreSQL wire protocol
> [1] to emulate Postgres server (Of course it includes emulating
> `pg_catalog` schema). It gives a user the ability to connect to a calcite
> schema using PostgreSQL tools and libraries.
>
> Our team is thinking to open source code of this project. And at first, we
> would like to know if anyone would be interested before doing any next
> steps.
>
> If you would be interested in, just write +1
>
> [1] https://www.postgresql.org/docs/current/protocol.html
>
Re: Does someone interested in integration calcite + PostgreSQL wire protocol ?
Posted by Mark Hammond <gp...@themarkhammond.com>.
+1
On Sat, 10 Sep 2022 at 23:29, Dmitry Sysolyatin <dm...@gmail.com>
wrote:
> Hello!
>
> I've been working on the integration calcite and PostgreSQL wire protocol
> [1] to emulate Postgres server (Of course it includes emulating
> `pg_catalog` schema). It gives a user the ability to connect to a calcite
> schema using PostgreSQL tools and libraries.
>
> Our team is thinking to open source code of this project. And at first, we
> would like to know if anyone would be interested before doing any next
> steps.
>
> If you would be interested in, just write +1
>
> [1] https://www.postgresql.org/docs/current/protocol.html
>