You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Tugdual Grall <tu...@gmail.com> on 2020/10/17 07:45:42 UTC
How to test new datasource without "in memory/mock" implementation
Hello,
I am new to the Calcite (at least trying to contribute - I did use it
indirectly a lot in Apache Drill )
I would like to enrich the Redis adapter (or create a new one) to support
RediSearch ( http://redisearch.io ) since it provides richer features such
as secondary indexing, querying and aggregations on data located in Redis.
What is the best way to run tests?
I see that the current Redis tests are using a fake implementation, same
for Mongo...
RediSearch does not have such an implementation (and same for other modules
Graph, JSON, ..)
Are we "allowed" to use containers in the tests, for example using
TestContainers ( https://www.testcontainers.org/ ) ?
If not can you point me to the proper way to run tests when a mock server
does not exist?
Thanks
Tug
Re: How to test new datasource without "in memory/mock" implementation
Posted by Andrei Sereda <an...@sereda.cc>.
I agree with Julian. Not everybody has the option to run docker locally.
Perhaps execute tests against embedded (or fake) instance as well as a real
one (docker or externally managed) ? Like it is done in
MongoDatabasePolicy.java [1]
Currently tests for Elasticsearch, Geode and Mongo are all run against
embedded test instances but one should be able to execute them against a
real one as well.
https://github.com/apache/calcite/blob/master/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoDatabasePolicy.java#L70
[1]
On Sun, Oct 18, 2020 at 3:30 AM Tugdual Grall <tu...@gmail.com> wrote:
> Ok
>
> I will look more into it
>
> On Sun, Oct 18, 2020 at 8:54 AM Julian Hyde <jh...@gmail.com>
> wrote:
>
> > This would be inconvenient for me personally. My company does not allow
> me
> > to run Docker on my work computer.
> >
> > Julian
> >
> > > On Oct 17, 2020, at 11:31 PM, Tugdual Grall <tu...@gmail.com> wrote:
> > >
> > > Hi
> > >
> > > I have created this PR to share with the community what could be done:
> > > * https://github.com/apache/calcite/pull/2223
> > >
> > > And associated issue
> https://issues.apache.org/jira/browse/CALCITE-4344
> > >
> > > I guess we will continue the discussion on the issue itself
> > >
> > > Tug
> > >
> > >> On Sat, Oct 17, 2020 at 11:05 PM Vladimir Sitnikov <
> > >> sitnikov.vladimir@gmail.com> wrote:
> > >>
> > >> Tugdual>Are we "allowed" to use containers in the tests, for example
> > >> using TestContainers
> > >>
> > >> I believe Testcontainers-based tests are more than welcome, however,
> > >> nobody implemented that yet.
> > >> It would be great if you could contribute that.
> > >>
> > >> Vladimir
> > >>
> > >>
> >
>
Re: How to test new datasource without "in memory/mock" implementation
Posted by Tugdual Grall <tu...@gmail.com>.
Ok
I will look more into it
On Sun, Oct 18, 2020 at 8:54 AM Julian Hyde <jh...@gmail.com> wrote:
> This would be inconvenient for me personally. My company does not allow me
> to run Docker on my work computer.
>
> Julian
>
> > On Oct 17, 2020, at 11:31 PM, Tugdual Grall <tu...@gmail.com> wrote:
> >
> > Hi
> >
> > I have created this PR to share with the community what could be done:
> > * https://github.com/apache/calcite/pull/2223
> >
> > And associated issue https://issues.apache.org/jira/browse/CALCITE-4344
> >
> > I guess we will continue the discussion on the issue itself
> >
> > Tug
> >
> >> On Sat, Oct 17, 2020 at 11:05 PM Vladimir Sitnikov <
> >> sitnikov.vladimir@gmail.com> wrote:
> >>
> >> Tugdual>Are we "allowed" to use containers in the tests, for example
> >> using TestContainers
> >>
> >> I believe Testcontainers-based tests are more than welcome, however,
> >> nobody implemented that yet.
> >> It would be great if you could contribute that.
> >>
> >> Vladimir
> >>
> >>
>
Re: How to test new datasource without "in memory/mock" implementation
Posted by Julian Hyde <jh...@gmail.com>.
This would be inconvenient for me personally. My company does not allow me to run Docker on my work computer.
Julian
> On Oct 17, 2020, at 11:31 PM, Tugdual Grall <tu...@gmail.com> wrote:
>
> Hi
>
> I have created this PR to share with the community what could be done:
> * https://github.com/apache/calcite/pull/2223
>
> And associated issue https://issues.apache.org/jira/browse/CALCITE-4344
>
> I guess we will continue the discussion on the issue itself
>
> Tug
>
>> On Sat, Oct 17, 2020 at 11:05 PM Vladimir Sitnikov <
>> sitnikov.vladimir@gmail.com> wrote:
>>
>> Tugdual>Are we "allowed" to use containers in the tests, for example
>> using TestContainers
>>
>> I believe Testcontainers-based tests are more than welcome, however,
>> nobody implemented that yet.
>> It would be great if you could contribute that.
>>
>> Vladimir
>>
>>
Re: How to test new datasource without "in memory/mock" implementation
Posted by Tugdual Grall <tu...@gmail.com>.
Hi
I have created this PR to share with the community what could be done:
* https://github.com/apache/calcite/pull/2223
And associated issue https://issues.apache.org/jira/browse/CALCITE-4344
I guess we will continue the discussion on the issue itself
Tug
On Sat, Oct 17, 2020 at 11:05 PM Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:
> Tugdual>Are we "allowed" to use containers in the tests, for example
> using TestContainers
>
> I believe Testcontainers-based tests are more than welcome, however,
> nobody implemented that yet.
> It would be great if you could contribute that.
>
> Vladimir
>
>
Re: How to test new datasource without "in memory/mock" implementation
Posted by Vladimir Sitnikov <si...@gmail.com>.
Tugdual>Are we "allowed" to use containers in the tests, for example using
TestContainers
I believe Testcontainers-based tests are more than welcome, however, nobody
implemented that yet.
It would be great if you could contribute that.
Vladimir
Re: How to test new datasource without "in memory/mock" implementation
Posted by Julian Hyde <jh...@apache.org>.
We need a solution that everyone can use easily. Ideally, contributors
to the rest of Calcite - say Calcite core - should not have to spend
extra effort, or experience any extra delay or flakiness, in order to
verify that they have not broken the Redis adapter.
The nearest example we have today is the Druid adapter. Stamatis spent
considerable time recently [1] to set up a container for Druid,
document how to use it, and enable it in CI. Someone could do the same
for Redis, preferably using the same container technology.
Julian
[1] https://issues.apache.org/jira/browse/CALCITE-4278
On Sat, Oct 17, 2020 at 12:46 AM Tugdual Grall <tu...@gmail.com> wrote:
>
> Hello,
>
> I am new to the Calcite (at least trying to contribute - I did use it
> indirectly a lot in Apache Drill )
>
> I would like to enrich the Redis adapter (or create a new one) to support
> RediSearch ( http://redisearch.io ) since it provides richer features such
> as secondary indexing, querying and aggregations on data located in Redis.
>
> What is the best way to run tests?
> I see that the current Redis tests are using a fake implementation, same
> for Mongo...
>
> RediSearch does not have such an implementation (and same for other modules
> Graph, JSON, ..)
>
> Are we "allowed" to use containers in the tests, for example using
> TestContainers ( https://www.testcontainers.org/ ) ?
>
> If not can you point me to the proper way to run tests when a mock server
> does not exist?
>
> Thanks
> Tug