You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by "Yang, Jerry" <cy...@ebay.com> on 2016/06/28 15:45:48 UTC
Adapter for couchbase
Hi guys,
Just joined the mailing list. Please advise if this is the proper place for such discussion, thanks in advance.
My colleague and I are using calcite to develop our project, and we need to support couchbase as part of the requirement. I am wondering if couchbase adapter is on the roadmap, or it is not planned yet.
We are planning to support both couchbase 3 and 4, so if it is not on the roadmap, we might be able to contribute back to the project when we are done.
Currently I have some preliminary designs. The approach to couchbase would be different based on major version. In couchbase 4, since it supports n1ql, an implementation similar to Cassandra’s CQL might be feasible, whereas in couchbase 3, native API calls is required.
Thanks again for help and please let me know if there’s a couchbase adapter, otherwise we are planning to do it ourselves.
Thanks,
Jerry Yang
Re: Adapter for couchbase
Posted by Julian Hyde <jh...@apache.org>.
Welcome to the list, Jerry! Yes, this is the perfect place for such discussions.
We would love to have a Couchbase adapter as part of the project. I agree with Michael’s suggestion to start from an existing adapter.
I created a JIRA case: https://issues.apache.org/jira/browse/CALCITE-1303 <https://issues.apache.org/jira/browse/CALCITE-1303>. You can add design details there, when you know them.
I get confused about the relationship between Couchbase and Apache CouchDB. Would this adapter work for both, or just for Couchbase? Maybe you could clarify this in the JIRA case.
Most of the recent adapters have also contributed to the test VM https://github.com/vlsi/calcite-test-dataset <https://github.com/vlsi/calcite-test-dataset>, so that anyone can easily run your unit tests. Vladimir, who owns that project, is a Calcite PMC member, so you can ask questions on this list.
Julian
> On Jun 28, 2016, at 9:02 AM, Michael Mior <mm...@uwaterloo.ca> wrote:
>
> AFAIK there are not currently any plans to develop a Couchbase adapter. I
> think this would be a welcome addition to Calcite. The Cassandra and
> MongoDB adapters are probably a good place to start. It would be great if
> you would be willing to contribute this to the project when you're done.
> Feel free to ask here if you have any specific questions when you get
> started!
>
> --
> Michael Mior
> michael.mior@gmail.com
>
> 2016-06-28 11:45 GMT-04:00 Yang, Jerry <cy...@ebay.com>:
>
>> Hi guys,
>>
>> Just joined the mailing list. Please advise if this is the proper place
>> for such discussion, thanks in advance.
>>
>> My colleague and I are using calcite to develop our project, and we need
>> to support couchbase as part of the requirement. I am wondering if
>> couchbase adapter is on the roadmap, or it is not planned yet.
>>
>> We are planning to support both couchbase 3 and 4, so if it is not on the
>> roadmap, we might be able to contribute back to the project when we are
>> done.
>>
>> Currently I have some preliminary designs. The approach to couchbase
>> would be different based on major version. In couchbase 4, since it
>> supports n1ql, an implementation similar to Cassandra’s CQL might be
>> feasible, whereas in couchbase 3, native API calls is required.
>>
>> Thanks again for help and please let me know if there’s a couchbase
>> adapter, otherwise we are planning to do it ourselves.
>>
>> Thanks,
>> Jerry Yang
Re: Adapter for couchbase
Posted by Michael Mior <mm...@uwaterloo.ca>.
Jerry,
Great to see that you're making some progress. I'm not very familiar with
Couchbase, but it seems like there is a drastic difference in the data
model between versions 3 and 4. You may find it easier to create a separate
adapter to handle Couchbase 3 if supporting it is important to you. This
would solve points #1 and #2 since you could use views in Couchbase 3 and
N1QL in Couchbase 4. I'm not quite sure what you're proposing in point #3.
A more concrete example might be helpful.
Regarding point #4, I'd suggest not worrying about that for the time being.
If your adapter doesn't explicitly handle aggregation, you can have Calcite
handle this client-side for now. Performance will probably be relatively
poor, but this is something that can be fixed on the future once you have a
working adapter.
Hope some of this helps!
Cheers,
--
Michael Mior
michael.mior@gmail.com
2016-07-05 19:58 GMT-04:00 Yang, Jerry <je...@ebay.com>:
> Hi guys,
>
> As I started working on couchbase adapters, I’ve encountered some
> questions that I’d hope to clarify prior to further implementation.
>
> Couchbase currently supports two major versions: 3 and 4. In coucbase 4,
> N1QL, a SQL-like language is introduced. In N1QL, users can select some
> fields inside values (essentially the JSON document) from a bucket, much
> like what CQL of Cassandra has. It makes the implementation simple and
> neat as far as Calcite is concerned; most of the SQL from Calcite can yield
> similar mappings to N1QL; as long as primary index is created, “SELECT *
> from [bucket_name]” is possible.
>
> Couchbase 3, on the other hand, is not as simple. View in couchbase
> contains a mapper and a reducer, and a query is implemented via ‘view’, so
> fields needs to be part of view; queries containing outside fields wouldn’t
> work. Also, filters are done at the client level if not defined in view.
> See below for API example. (Java8; I’d probably implement this in guava
> since calcite does not support Java8 yet, but you get the idea.)
>
> bucket
> .async()
> .query(ViewQuery.from("beers_and_breweries", "by_name"))
> .flatMap(AsyncViewResult::rows)
> .flatMap(AsyncViewRow::document)
> .filter(doc -> doc.content().getString("type").equals("beer"))
> .subscribe(doc -> System.out.println(doc.content().getString("name")));
> (http://docs.couchbase.com/developer/java-2.1/java-intro.html)
>
> So here’smy take away.
>
> 1. It seems to me it makes more sense to do “SELECT * from
> [view_name]” instead of bucket name; view contains set of fields to select
> from. Yet it is NOT compatible with couchbase 4 and in general not a good
> idea.
>
> 2. Should I choose to use ‘bucket’ as table, I then have to predefine
> bucket name as well as view name in my model, which makes ‘FROM’ obsolete
> since it doesn’t consider “FROM” field anymore.
>
> 3. Potentially we could have a table schema as part of the model,
> much like mongodb, and then dynamic create view(s) based on table schema.
> However, it is very invasive and I am not sure of the implication of the
> performance. Also, it requires quite some work since it means a schema to
> mapper/reducer transformer.
>
> 4. With aggregation like distinct or count, it would have to be done
> via client side aggregation if it is not part of the view; performance
> might be poor without reducer.
>
> Any thoughts would be greatly appreciated.
>
> Thanks,
> Jerry
>
>
> On 6/28/16, 9:02 AM, "michael.mior@gmail.com on behalf of Michael Mior" <
> michael.mior@gmail.com on behalf of mmior@uwaterloo.ca> wrote:
>
> AFAIK there are not currently any plans to develop a Couchbase adapter. I
> think this would be a welcome addition to Calcite. The Cassandra and
> MongoDB adapters are probably a good place to start. It would be great if
> you would be willing to contribute this to the project when you're done.
> Feel free to ask here if you have any specific questions when you get
> started!
>
> --
> Michael Mior
> michael.mior@gmail.com
>
> 2016-06-28 11:45 GMT-04:00 Yang, Jerry <cy...@ebay.com>:
>
> > Hi guys,
> >
> > Just joined the mailing list. Please advise if this is the proper place
> > for such discussion, thanks in advance.
> >
> > My colleague and I are using calcite to develop our project, and we need
> > to support couchbase as part of the requirement. I am wondering if
> > couchbase adapter is on the roadmap, or it is not planned yet.
> >
> > We are planning to support both couchbase 3 and 4, so if it is not on the
> > roadmap, we might be able to contribute back to the project when we are
> > done.
> >
> > Currently I have some preliminary designs. The approach to couchbase
> > would be different based on major version. In couchbase 4, since it
> > supports n1ql, an implementation similar to Cassandra’s CQL might be
> > feasible, whereas in couchbase 3, native API calls is required.
> >
> > Thanks again for help and please let me know if there’s a couchbase
> > adapter, otherwise we are planning to do it ourselves.
> >
> > Thanks,
> > Jerry Yang
>
>
>
Re: Adapter for couchbase
Posted by "Yang, Jerry" <je...@ebay.com>.
Hi guys,
As I started working on couchbase adapters, I’ve encountered some questions that I’d hope to clarify prior to further implementation.
Couchbase currently supports two major versions: 3 and 4. In coucbase 4, N1QL, a SQL-like language is introduced. In N1QL, users can select some fields inside values (essentially the JSON document) from a bucket, much like what CQL of Cassandra has. It makes the implementation simple and neat as far as Calcite is concerned; most of the SQL from Calcite can yield similar mappings to N1QL; as long as primary index is created, “SELECT * from [bucket_name]” is possible.
Couchbase 3, on the other hand, is not as simple. View in couchbase contains a mapper and a reducer, and a query is implemented via ‘view’, so fields needs to be part of view; queries containing outside fields wouldn’t work. Also, filters are done at the client level if not defined in view. See below for API example. (Java8; I’d probably implement this in guava since calcite does not support Java8 yet, but you get the idea.)
bucket
.async()
.query(ViewQuery.from("beers_and_breweries", "by_name"))
.flatMap(AsyncViewResult::rows)
.flatMap(AsyncViewRow::document)
.filter(doc -> doc.content().getString("type").equals("beer"))
.subscribe(doc -> System.out.println(doc.content().getString("name")));
(http://docs.couchbase.com/developer/java-2.1/java-intro.html)
So here’smy take away.
1. It seems to me it makes more sense to do “SELECT * from [view_name]” instead of bucket name; view contains set of fields to select from. Yet it is NOT compatible with couchbase 4 and in general not a good idea.
2. Should I choose to use ‘bucket’ as table, I then have to predefine bucket name as well as view name in my model, which makes ‘FROM’ obsolete since it doesn’t consider “FROM” field anymore.
3. Potentially we could have a table schema as part of the model, much like mongodb, and then dynamic create view(s) based on table schema. However, it is very invasive and I am not sure of the implication of the performance. Also, it requires quite some work since it means a schema to mapper/reducer transformer.
4. With aggregation like distinct or count, it would have to be done via client side aggregation if it is not part of the view; performance might be poor without reducer.
Any thoughts would be greatly appreciated.
Thanks,
Jerry
On 6/28/16, 9:02 AM, "michael.mior@gmail.com on behalf of Michael Mior" <michael.mior@gmail.com on behalf of mmior@uwaterloo.ca> wrote:
AFAIK there are not currently any plans to develop a Couchbase adapter. I
think this would be a welcome addition to Calcite. The Cassandra and
MongoDB adapters are probably a good place to start. It would be great if
you would be willing to contribute this to the project when you're done.
Feel free to ask here if you have any specific questions when you get
started!
--
Michael Mior
michael.mior@gmail.com
2016-06-28 11:45 GMT-04:00 Yang, Jerry <cy...@ebay.com>:
> Hi guys,
>
> Just joined the mailing list. Please advise if this is the proper place
> for such discussion, thanks in advance.
>
> My colleague and I are using calcite to develop our project, and we need
> to support couchbase as part of the requirement. I am wondering if
> couchbase adapter is on the roadmap, or it is not planned yet.
>
> We are planning to support both couchbase 3 and 4, so if it is not on the
> roadmap, we might be able to contribute back to the project when we are
> done.
>
> Currently I have some preliminary designs. The approach to couchbase
> would be different based on major version. In couchbase 4, since it
> supports n1ql, an implementation similar to Cassandra’s CQL might be
> feasible, whereas in couchbase 3, native API calls is required.
>
> Thanks again for help and please let me know if there’s a couchbase
> adapter, otherwise we are planning to do it ourselves.
>
> Thanks,
> Jerry Yang
Re: Adapter for couchbase
Posted by "Yang, Jerry" <je...@ebay.com>.
Hi Michael,
Thanks for the advice and sorry for the delayed response. I’ve made more progress by using view as part of the assumption. Will need to document more before release so that it would be a painless onboarding process.
During the filter development, I’ve encountered some issues regarding LogicalFilter. Since couchbase3 has very limited support on value filter (as JsonDocument), most of the filter capability relies on LogicalFilter. However, once I have my own filter to specifically get the document_id as predicate to pass them along to the query phase, I am not able to failback to LogicalFilter. Is there an elegant way to support such feature? I would like to only invoke my filter for some specific predicates.
I am wrapping the couchbase3 code with some more unit test cases. Will need to follow my company’s process before I can finally contribute back to calcite.
-Jerry
Re: Adapter for couchbase
Posted by Michael Mior <mm...@uwaterloo.ca>.
AFAIK there are not currently any plans to develop a Couchbase adapter. I
think this would be a welcome addition to Calcite. The Cassandra and
MongoDB adapters are probably a good place to start. It would be great if
you would be willing to contribute this to the project when you're done.
Feel free to ask here if you have any specific questions when you get
started!
--
Michael Mior
michael.mior@gmail.com
2016-06-28 11:45 GMT-04:00 Yang, Jerry <cy...@ebay.com>:
> Hi guys,
>
> Just joined the mailing list. Please advise if this is the proper place
> for such discussion, thanks in advance.
>
> My colleague and I are using calcite to develop our project, and we need
> to support couchbase as part of the requirement. I am wondering if
> couchbase adapter is on the roadmap, or it is not planned yet.
>
> We are planning to support both couchbase 3 and 4, so if it is not on the
> roadmap, we might be able to contribute back to the project when we are
> done.
>
> Currently I have some preliminary designs. The approach to couchbase
> would be different based on major version. In couchbase 4, since it
> supports n1ql, an implementation similar to Cassandra’s CQL might be
> feasible, whereas in couchbase 3, native API calls is required.
>
> Thanks again for help and please let me know if there’s a couchbase
> adapter, otherwise we are planning to do it ourselves.
>
> Thanks,
> Jerry Yang