Reasoning about removal of OpProcessors in TP 4.0

[Andrii Lomakin <an...@jetbrains.com.INVALID>]

Good day.

Could you clarify the reasoning behind the removal of OpProcessors support
in TP 4.0 and what extension mechanics are proposed as an alternative?

Example of how we use OpProcessors for protocol extensions:

1. Extension of core OpProcessors - during TX processing, IDs of our
elements undergo two phases: temporary and permanent. During which their
values are changed. Because query results are streamed, we send additional
metadata that maps temporary IDs to the permanent ones to keep client data
in sync in the final response Frame.
2. We provide our own OpProcessor, which handles commands related to the
management of database instances present on the server.

Which mechanics are provided for such and similar extension points in TP 4?

Re: Reasoning about removal of OpProcessors in TP 4.0

[Andrii Lomakin <an...@jetbrains.com.INVALID>]

>  It will be nice to have you as part of
that discussion.

Thank you, I am only pro for this.

On Tue, Nov 11, 2025 at 2:58 PM Stephen Mallette <sp...@gmail.com>
wrote:

> Most core contributors moved to working on 3.8.0 earlier this year so
> ideas/discussions for 4.x stalled a bit. Now that 3.8.0 is
> nearing completion I suspect that 4.x talk will begin again. There is a lot
> of focus on driver/server interaction for 4.x so this notion of extension
> for providers will come up soon. It will be nice to have you as part of
> that discussion.
>
> On Mon, Nov 10, 2025 at 10:13 AM Andrii Lomakin
> <an...@jetbrains.com.invalid> wrote:
>
> > Hi Sephen,
> > Thank you for your response.
> >
> > In such a case, I suppose I will provide a PR that makes the extension of
> > OpProcessors a relatively easy task and provide several use cases in the
> PR
> > and as a blog post.
> > Hopefully, they will be useful.
> >
> > As for TP 4, I would appreciate it if you could keep us posted, if you
> > don't mind.
> >
> >
> > On Mon, Nov 10, 2025 at 4:09 PM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > I'm not sure that there have been many successful extensions of
> > OpProcessor
> > > (though you describe yours as doing something for you). I've mostly
> seen
> > > that providers have found the extensions points insufficient for their
> > > needs and end up in a fork or at the netty level. There's something not
> > > right there in their abstraction for most cases. I think with the move
> to
> > > pure HTTP in 4.x it made sense to divest the server of some of the
> layers
> > > it had. Ultimately, I think extension points for the server in 4.x are
> up
> > > for discussion. afaik, no decisions have been made as to what the
> > > alternative will be.
> > >
> > > On Mon, Nov 10, 2025 at 3:17 AM Andrii Lomakin
> > > <an...@jetbrains.com.invalid> wrote:
> > >
> > > > Good day.
> > > >
> > > > Could you clarify the reasoning behind the removal of OpProcessors
> > > support
> > > > in TP 4.0 and what extension mechanics are proposed as an
> alternative?
> > > >
> > > > Example of how we use OpProcessors for protocol extensions:
> > > >
> > > > 1. Extension of core OpProcessors - during TX processing, IDs of our
> > > > elements undergo two phases: temporary and permanent. During which
> > their
> > > > values are changed. Because query results are streamed, we send
> > > additional
> > > > metadata that maps temporary IDs to the permanent ones to keep client
> > > data
> > > > in sync in the final response Frame.
> > > > 2. We provide our own OpProcessor, which handles commands related to
> > the
> > > > management of database instances present on the server.
> > > >
> > > > Which mechanics are provided for such and similar extension points in
> > TP
> > > 4?
> > > >
> > >
> >
>

Re: Reasoning about removal of OpProcessors in TP 4.0

[Stephen Mallette <sp...@gmail.com>]

Most core contributors moved to working on 3.8.0 earlier this year so
ideas/discussions for 4.x stalled a bit. Now that 3.8.0 is
nearing completion I suspect that 4.x talk will begin again. There is a lot
of focus on driver/server interaction for 4.x so this notion of extension
for providers will come up soon. It will be nice to have you as part of
that discussion.

On Mon, Nov 10, 2025 at 10:13 AM Andrii Lomakin
<an...@jetbrains.com.invalid> wrote:

> Hi Sephen,
> Thank you for your response.
>
> In such a case, I suppose I will provide a PR that makes the extension of
> OpProcessors a relatively easy task and provide several use cases in the PR
> and as a blog post.
> Hopefully, they will be useful.
>
> As for TP 4, I would appreciate it if you could keep us posted, if you
> don't mind.
>
>
> On Mon, Nov 10, 2025 at 4:09 PM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > I'm not sure that there have been many successful extensions of
> OpProcessor
> > (though you describe yours as doing something for you). I've mostly seen
> > that providers have found the extensions points insufficient for their
> > needs and end up in a fork or at the netty level. There's something not
> > right there in their abstraction for most cases. I think with the move to
> > pure HTTP in 4.x it made sense to divest the server of some of the layers
> > it had. Ultimately, I think extension points for the server in 4.x are up
> > for discussion. afaik, no decisions have been made as to what the
> > alternative will be.
> >
> > On Mon, Nov 10, 2025 at 3:17 AM Andrii Lomakin
> > <an...@jetbrains.com.invalid> wrote:
> >
> > > Good day.
> > >
> > > Could you clarify the reasoning behind the removal of OpProcessors
> > support
> > > in TP 4.0 and what extension mechanics are proposed as an alternative?
> > >
> > > Example of how we use OpProcessors for protocol extensions:
> > >
> > > 1. Extension of core OpProcessors - during TX processing, IDs of our
> > > elements undergo two phases: temporary and permanent. During which
> their
> > > values are changed. Because query results are streamed, we send
> > additional
> > > metadata that maps temporary IDs to the permanent ones to keep client
> > data
> > > in sync in the final response Frame.
> > > 2. We provide our own OpProcessor, which handles commands related to
> the
> > > management of database instances present on the server.
> > >
> > > Which mechanics are provided for such and similar extension points in
> TP
> > 4?
> > >
> >
>

Re: Reasoning about removal of OpProcessors in TP 4.0

[Andrii Lomakin <an...@jetbrains.com.INVALID>]

Hi Sephen,
Thank you for your response.

In such a case, I suppose I will provide a PR that makes the extension of
OpProcessors a relatively easy task and provide several use cases in the PR
and as a blog post.
Hopefully, they will be useful.

As for TP 4, I would appreciate it if you could keep us posted, if you
don't mind.


On Mon, Nov 10, 2025 at 4:09 PM Stephen Mallette <sp...@gmail.com>
wrote:

> I'm not sure that there have been many successful extensions of OpProcessor
> (though you describe yours as doing something for you). I've mostly seen
> that providers have found the extensions points insufficient for their
> needs and end up in a fork or at the netty level. There's something not
> right there in their abstraction for most cases. I think with the move to
> pure HTTP in 4.x it made sense to divest the server of some of the layers
> it had. Ultimately, I think extension points for the server in 4.x are up
> for discussion. afaik, no decisions have been made as to what the
> alternative will be.
>
> On Mon, Nov 10, 2025 at 3:17 AM Andrii Lomakin
> <an...@jetbrains.com.invalid> wrote:
>
> > Good day.
> >
> > Could you clarify the reasoning behind the removal of OpProcessors
> support
> > in TP 4.0 and what extension mechanics are proposed as an alternative?
> >
> > Example of how we use OpProcessors for protocol extensions:
> >
> > 1. Extension of core OpProcessors - during TX processing, IDs of our
> > elements undergo two phases: temporary and permanent. During which their
> > values are changed. Because query results are streamed, we send
> additional
> > metadata that maps temporary IDs to the permanent ones to keep client
> data
> > in sync in the final response Frame.
> > 2. We provide our own OpProcessor, which handles commands related to the
> > management of database instances present on the server.
> >
> > Which mechanics are provided for such and similar extension points in TP
> 4?
> >
>

Re: Reasoning about removal of OpProcessors in TP 4.0

[Stephen Mallette <sp...@gmail.com>]

I'm not sure that there have been many successful extensions of OpProcessor
(though you describe yours as doing something for you). I've mostly seen
that providers have found the extensions points insufficient for their
needs and end up in a fork or at the netty level. There's something not
right there in their abstraction for most cases. I think with the move to
pure HTTP in 4.x it made sense to divest the server of some of the layers
it had. Ultimately, I think extension points for the server in 4.x are up
for discussion. afaik, no decisions have been made as to what the
alternative will be.

On Mon, Nov 10, 2025 at 3:17 AM Andrii Lomakin
<an...@jetbrains.com.invalid> wrote:

> Good day.
>
> Could you clarify the reasoning behind the removal of OpProcessors support
> in TP 4.0 and what extension mechanics are proposed as an alternative?
>
> Example of how we use OpProcessors for protocol extensions:
>
> 1. Extension of core OpProcessors - during TX processing, IDs of our
> elements undergo two phases: temporary and permanent. During which their
> values are changed. Because query results are streamed, we send additional
> metadata that maps temporary IDs to the permanent ones to keep client data
> in sync in the final response Frame.
> 2. We provide our own OpProcessor, which handles commands related to the
> management of database instances present on the server.
>
> Which mechanics are provided for such and similar extension points in TP 4?
>