You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by "Gupta, Shuchi" <Sh...@ca.com> on 2016/12/27 11:26:50 UTC
Eliminating model.json from druid implementation
Hi Team
In reference to apache druid, I have been using a JSON file as metadata for my connections to a custom database based on elastic search.
Instead I intend to create the metadata on the fly by eliminating this json path from the connection url and use :
jdbc:calcite:schemaFactory=org.apache.calcite.adapter.druid.DruidSchemaFactory; schema.url=http://localhost:8082; schema.coordinatorUrl=http://localhost:8081
The connection is passing successfully, but I fail to understand where shall I map the schema details . Kindly let me know what change would be required to implement this functionality.
Thanks and Regards
Shuchi Gupta
Software Engineer (CA)
CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
Office: +71262 | Shuchi.Gupta2@ca.com
<ma...@ca.com>[CA]<http://www.ca.com/us/default.aspx>[Twitter]<http://twitter.com/CAInc>[Slideshare]<http://www.slideshare.net/cainc>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Google+]<https://plus.google.com/+CATechnologies>[Google+]<http://www.ca.com/us/rss.aspx?intcmp=footernav>
RE: Eliminating model.json from druid implementation
Posted by "Gupta, Shuchi" <Sh...@ca.com>.
Hello Julian
My use case is :
1. Database is based on elastic search
2. When I configure the data source using the syntax :
jdbc:calcite:schemaFactory=org.apache.calcite.adapter.druid.DruidSchemaFactory; schema.host=http://localhost:8082; schema.urlpath=/das/<tenantid>/<prodid>/<docid> , the connection is successful. (Though it gets no tables here as the schema details are not passed)
3. However when I try to query on this database, an exception is thrown, since the schema is not found.
My custom database also has an API to return the schema details in the format :
{
"mappings": {
"some_type": {
"properties": {
"reqSize": {
"index": "no",
"store": "yes",
"type": "integer"
},
"apiName": {
"index": "not_analyzed",
"store": "yes",
"type": "string"
},
"accountPlanName": {
"index": "not_analyzed",
"store": "yes",
"type": "string"
}
I need to understand how(which calcite classes) to leverage in order to suffice the details which the driver expects from the model.json viz. dimensions, ,metrics, datasource.
I referred to http://druid.io/docs/latest/querying/segmentmetadataquery.html (which in druid, discovers the columns of the table), but again could not find the required context.
Thanks and Regards
Shuchi Gupta
Software Engineer (CA)
-----Original Message-----
From: Julian Hyde [mailto:jhyde@apache.org]
Sent: Tuesday, December 27, 2016 11:18 PM
To: dev@calcite.apache.org
Subject: Re: Eliminating model.json from druid implementation
We introduced the “schemaFactory” connect string parameter in https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CALCITE-2D1259&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=CoSEy4c4OzsxPsEiEpC6AXlB_QpnqxVwx1oi1kaQg50&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CALCITE-2D1259&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=CoSEy4c4OzsxPsEiEpC6AXlB_QpnqxVwx1oi1kaQg50&e= > for cases where the connection had a single schema and that schema was trivial. That doesn’t mean that the schema is empty (a schema with no tables is fairly useless) but that it is self-populating. If I recall correctly the Druid schema factory creates a Calcite table for each data set in Druid, so it fulfills these criteria.
But if you need more content in a schema, I suggest you go back to the old way and generate a small JSON model. Note that you can prefix the model URL with “inline:” to pass the content of the model rather than putting the model in a temporary file.
Another alternative is to dispense with the model entirely and use methods such as Schema.addTable to build the schema programmatically. Really, the JSON model stuff is just a builder on top of that.
Julian
> On Dec 27, 2016, at 3:26 AM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
> Hi Team
>
> In reference to apache druid, I have been using a JSON file as metadata for my connections to a custom database based on elastic search.
> Instead I intend to create the metadata on the fly by eliminating this json path from the connection url and use :
> jdbc:calcite:schemaFactory=org.apache.calcite.adapter.druid.DruidSchemaFactory; schema.url=https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8082&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=OycHaZLAMQQghbn3mXboRMji1wCxmFjLZX0D0rUBJ7Y&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8082_&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=T49R90tuqZBfVBSSzLcajDsMvEnSqSVZmnNncepsPJI&e= >; schema.coordinatorUrl=https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8081&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=wzgfGP9HYu8RhR3vFxpQVRCmLpYr2oyXevKyGKAyfEk&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8081_&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=_yVux-ThDVg6cWxvPKimL3fH2aFpDu9GLTOscJtp7t8&e= >
>
> The connection is passing successfully, but I fail to understand where shall I map the schema details . Kindly let me know what change would be required to implement this functionality.
>
>
> Thanks and Regards
> Shuchi Gupta
> Software Engineer (CA)
>
> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
> Office: +71262 | Shuchi.Gupta2@ca.com
>
> <ma...@ca.com> <http://www.ca.com/us/default.aspx> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_CAInc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=JzFK044MbEhXHelmPYjCCOG92mSJHFu4gATLhvNhPY0&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_cainc&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=NrO13bz7BJYqUDx5nY_M6RGEDF68hsTP32_SfLXrIRs&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_CATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=ks2-feP5jzgKs3wj_H2ruahrqErjryRkxgOkNudFYfQ&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_user_catechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=ZjmZaqfCaSLgP7im_zKfexZrFGiTWKxKWdE48Ypm3ko&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_1372-3Fgoback-3D.cps-5F1244823420724-5F1&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=TxJ7K8VMK8PRSxyfmZbRAaIBm9Nonur5wgpb7Im9KMo&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BCATechnologies&d=DwIFaQ&c=_hRq4mqlUmqpqlyQ5hkoDXIVh6I6pxfkkNxQuL0p-Z0&r=_Fj_BSkIJErzZ82vma0tSpR2D2tntATQeeTvZFPXEMI&m=-Z_7HdytOg0UWaKQGuutkRhOfJSOU1rbK5vlgxiNCac&s=x9K_bIo6jzauz7edv2y8wjj9U29y5ec3vxtyqoyxpTY&e= > <http://www.ca.com/us/rss.aspx?intcmp=footernav>
Re: Eliminating model.json from druid implementation
Posted by Julian Hyde <jh...@apache.org>.
We introduced the “schemaFactory” connect string parameter in https://issues.apache.org/jira/browse/CALCITE-1259 <https://issues.apache.org/jira/browse/CALCITE-1259> for cases where the connection had a single schema and that schema was trivial. That doesn’t mean that the schema is empty (a schema with no tables is fairly useless) but that it is self-populating. If I recall correctly the Druid schema factory creates a Calcite table for each data set in Druid, so it fulfills these criteria.
But if you need more content in a schema, I suggest you go back to the old way and generate a small JSON model. Note that you can prefix the model URL with “inline:” to pass the content of the model rather than putting the model in a temporary file.
Another alternative is to dispense with the model entirely and use methods such as Schema.addTable to build the schema programmatically. Really, the JSON model stuff is just a builder on top of that.
Julian
> On Dec 27, 2016, at 3:26 AM, Gupta, Shuchi <Sh...@ca.com> wrote:
>
> Hi Team
>
> In reference to apache druid, I have been using a JSON file as metadata for my connections to a custom database based on elastic search.
> Instead I intend to create the metadata on the fly by eliminating this json path from the connection url and use :
> jdbc:calcite:schemaFactory=org.apache.calcite.adapter.druid.DruidSchemaFactory; schema.url=http://localhost:8082 <http://localhost:8082/>; schema.coordinatorUrl=http://localhost:8081 <http://localhost:8081/>
>
> The connection is passing successfully, but I fail to understand where shall I map the schema details . Kindly let me know what change would be required to implement this functionality.
>
>
> Thanks and Regards
> Shuchi Gupta
> Software Engineer (CA)
>
> CA Technologies | Plot 115/1, 115/16-115/21 | Hyderabad, 500032
> Office: +71262 | Shuchi.Gupta2@ca.com
>
> <ma...@ca.com> <http://www.ca.com/us/default.aspx> <http://twitter.com/CAInc> <http://www.slideshare.net/cainc> <https://www.facebook.com/CATechnologies> <http://www.youtube.com/user/catechnologies> <http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1> <https://plus.google.com/+CATechnologies> <http://www.ca.com/us/rss.aspx?intcmp=footernav>