Calcite Streaming and Mongo

[Andrei Sereda <an...@sereda.cc>]

Hello,

Since version 3.6 mongo introduced collection change streams
<https://docs.mongodb.com/v3.6/changeStreams/> which allow to observe
real-time changes of a collection. This looks very accommodating to calcite
streams <https://calcite.apache.org/docs/stream.html> which provides more
complex windowing and grouping operations.

Do you think it makes sense to support streams in Mongo Adapter ? Would you
be interested in such feature ?

Thanks in advance,
Andrei.

Re: Calcite Streaming and Mongo

[Shuyi Chen <su...@gmail.com>]

That will be cool. It can also be used as an example for other projects to
use Calcite streaming. I would imagine you need to translate the SQL
through the calcite logical plan to a set of pipeline operators and
pipeline stages in mongo. You can refer to projects like Flink, Samza or
Storm for referenece. Thanks.

Shuyi

On Wed, Oct 3, 2018 at 12:16 PM Michael Mior <mm...@apache.org> wrote:

> That would be great and an awesome example of an adapter using streaming. I
> haven't looked closely at Mongo's change streams, but you'd have to see if
> the semantics of the stream are suitable for exposing existing tables also
> as streams or if it would make more sense to separate streams and tables.
>
> --
> Michael Mior
> mmior@apache.org
>
>
> Le mar. 2 oct. 2018 à 20:43, Andrei Sereda <an...@sereda.cc> a écrit :
>
> > Hello,
> >
> > Since version 3.6 mongo introduced collection change streams
> > <https://docs.mongodb.com/v3.6/changeStreams/> which allow to observe
> > real-time changes of a collection. This looks very accommodating to
> calcite
> > streams <https://calcite.apache.org/docs/stream.html> which provides
> more
> > complex windowing and grouping operations.
> >
> > Do you think it makes sense to support streams in Mongo Adapter ? Would
> you
> > be interested in such feature ?
> >
> > Thanks in advance,
> > Andrei.
> >
>


-- 
"So you have to trust that the dots will somehow connect in your future."

Re: Calcite Streaming and Mongo

[Michael Mior <mm...@apache.org>]

That would be great and an awesome example of an adapter using streaming. I
haven't looked closely at Mongo's change streams, but you'd have to see if
the semantics of the stream are suitable for exposing existing tables also
as streams or if it would make more sense to separate streams and tables.

--
Michael Mior
mmior@apache.org


Le mar. 2 oct. 2018 à 20:43, Andrei Sereda <an...@sereda.cc> a écrit :

> Hello,
>
> Since version 3.6 mongo introduced collection change streams
> <https://docs.mongodb.com/v3.6/changeStreams/> which allow to observe
> real-time changes of a collection. This looks very accommodating to calcite
> streams <https://calcite.apache.org/docs/stream.html> which provides more
> complex windowing and grouping operations.
>
> Do you think it makes sense to support streams in Mongo Adapter ? Would you
> be interested in such feature ?
>
> Thanks in advance,
> Andrei.
>

Re: Calcite Streaming and Mongo

[Julian Hyde <jh...@apache.org>]

That would be very cool. MongoTable could implement StreamableTable, so queries that want to read change streams could do so.

Please log a JIRA case.

Julian


> On Oct 2, 2018, at 5:42 PM, Andrei Sereda <an...@sereda.cc> wrote:
> 
> Hello,
> 
> Since version 3.6 mongo introduced collection change streams
> <https://docs.mongodb.com/v3.6/changeStreams/> which allow to observe
> real-time changes of a collection. This looks very accommodating to calcite
> streams <https://calcite.apache.org/docs/stream.html> which provides more
> complex windowing and grouping operations.
> 
> Do you think it makes sense to support streams in Mongo Adapter ? Would you
> be interested in such feature ?
> 
> Thanks in advance,
> Andrei.