You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by LogSplitter <lo...@gmail.com> on 2017/07/04 07:03:27 UTC
Re: Question about CalciteSchema in 1.12
Hi,
I have been trying to get planner to work and it is really close but I
seem to get into issues due to one special requirement I have where I
have to remove a system column on 'select *' statements.
Background, we used to remove the special columns by processing
through the SqlNodeLIst of the validated query and rebuild the select
list (setSelectList()) with the system column I wanted removed. This
approach doesn't seem to work in the planner world as I believe I need
to do a validate after the selectList is modified but the planner model
doesn't like it when you don't do all steps exactly as it expected
(these types of errors Exception occurred: cannot move to STATE_3_PARSED
from STATE_4_VALIDATED)
I asked in a different thread how to handle the system columns and was
told to try making my own SqlToRelConverter.getSystemFields() but I
could not see how I could plug my own SqlToRelConverter into the planner
model.
So currently I am scratching my head stuck on an old version of calcite
wonder which direction I should head. Ideally I would like to use the
Planner model but need a little pointer of how I get a custom
SqlToRelConverter.getSystemFields() hooked up.
thanks
On 04/22/2017 10:07 AM, Julian Hyde wrote:
> Have you tried org.apache.calcite.tools.Planner? There are examples in org.apache.calcite.tools.PlannerTest. Barely a mention of SqlValidator, let alone CalciteSchema.
>
>
>> On Apr 21, 2017, at 10:17 PM, LogSplitter <lo...@gmail.com> wrote:
>>
>> Julian,
>>
>> So I am thinking that I am probably missing something here as I seem to need a CatalogReader to be able to get a SqlValidatorImpl to enable me to call validate to get the relational algebra. What would be the recommended way? Pointing me to a sample in the code base should get me on the right path as I may be way off by the sounds of it, even though it seems to work very well for what I need in 1.11.
>>
>> thanks
>>
>>
>> On 04/21/2017 09:32 PM, Julian Hyde wrote:
>>> MockCatalogReader was created for testing, but we don’t intend people to use it (or a modified version of it) in projects that use Calcite.
>>>
>>> CalciteSchema is a private class (the Java class is marked “public” only because it is used across several packages) and we don’t intend people to use it.
>>>
>>> Our intended extension point is Schema. Write your own instance of that API to go and get table definitions on demand. You need to write a getTableNames() method that returns all table names as a set, but you can create the table definitions one at a time when getTable(String name) is called.
>>>
>>> In https://issues.apache.org/jira/browse/CALCITE-1742 <https://issues.apache.org/jira/browse/CALCITE-1742> we are discussing possibly changing this model, but hopefully in a compatible way.
>>>
>>> Julian
>>>
>>>
>>>> On Apr 21, 2017, at 8:19 PM, LogSplitter <lo...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am just in the process of updating to the latest version of calcite for my project and have run into an issue and not sure whether I am missing something.
>>>>
>>>> I currently use a process based on some of the code around MockCatalogReader to create a catalog and allow my sql to get parsed to relational algebra, which is all I use from the calcite side of things.
>>>>
>>>> I was able to dynamically look up metadata previously in the code by having the CatalogReader getTable code lazily load the metadata items as it received request for tables.
>>>>
>>>> It seems with the 1.12 code and the requirement to use calciteSchema it expects all the schemas and tables to be available all the time.
>>>>
>>>> I am wondering if my lazy load approach is a mistake or if I should work on the calciteSchema interface to allow it to continue with this behaviour.
>>>>
>>>> Does anyone have thoughts on if what I am trying to do makes sense.
>>>>
>>>> thanks
>>>>
>>>>
Re: Question about CalciteSchema in 1.12
Posted by Julian Hyde <jh...@apache.org>.
Glad you noticed CALCITE-1849. (Nice coincidence that Gian ran into the same problem!) I am reviewing and it will be in shortly.
> On Jul 4, 2017, at 7:11 PM, LogSplitter <lo...@gmail.com> wrote:
>
> Julian,
>
> Thanks for your reply.
>
> I muddled through and built a planner based on the exact code as Planner for now with the config I needed.
>
> I did notice someone else had same issues and has built a PR with ability to pass a config, here https://issues.apache.org/jira/browse/CALCITE-1849, once this one lands we should be good.
>
> I like the system field config idea, I will try getting that working.
>
> Thanks
>
>
>
> On 07/04/2017 06:17 PM, Julian Hyde wrote:
>> Planner doesn’t support what you need right now. Can you log a JIRA case please?
>>
>> You don’t need to sub-class SqlToRelConverter to control its behavior, just pass in a SqlToRelConverter.Config that says exactly what its behavior should be. I see that PlannerImpl.rel builds a config but doesn’t allow it to be customized.
>>
>> Also change SqlToRelConverter.getSystemFields() so that it gets system fields from a member variable, populated from Config:
>>
>> public interface Config {
>> Function<RelDataTypeFactory, List<RelDataTypeField>> systemFieldsFactory();
>> }
>>
>> SqlToRelConverter() {
>> this.systemFieldList =
>> ImmutableList.copyOf(config.systemFieldsFactory().apply(typeFactory));
>> }
>>
>> Julian
>>
>>
>>> On Jul 4, 2017, at 12:03 AM, LogSplitter <lo...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I have been trying to get planner to work and it is really close but I seem to get into issues due to one special requirement I have where I have to remove a system column on 'select *' statements.
>>>
>>> Background, we used to remove the special columns by processing through the SqlNodeLIst of the validated query and rebuild the select list (setSelectList()) with the system column I wanted removed. This approach doesn't seem to work in the planner world as I believe I need to do a validate after the selectList is modified but the planner model doesn't like it when you don't do all steps exactly as it expected (these types of errors Exception occurred: cannot move to STATE_3_PARSED from STATE_4_VALIDATED)
>>>
>>> I asked in a different thread how to handle the system columns and was told to try making my own SqlToRelConverter.getSystemFields() but I could not see how I could plug my own SqlToRelConverter into the planner model.
>>>
>>> So currently I am scratching my head stuck on an old version of calcite wonder which direction I should head. Ideally I would like to use the Planner model but need a little pointer of how I get a custom SqlToRelConverter.getSystemFields() hooked up.
>>>
>>> thanks
>>>
>>>
>>> On 04/22/2017 10:07 AM, Julian Hyde wrote:
>>>> Have you tried org.apache.calcite.tools.Planner? There are examples in org.apache.calcite.tools.PlannerTest. Barely a mention of SqlValidator, let alone CalciteSchema.
>>>>
>>>>
>>>>> On Apr 21, 2017, at 10:17 PM, LogSplitter <lo...@gmail.com> wrote:
>>>>>
>>>>> Julian,
>>>>>
>>>>> So I am thinking that I am probably missing something here as I seem to need a CatalogReader to be able to get a SqlValidatorImpl to enable me to call validate to get the relational algebra. What would be the recommended way? Pointing me to a sample in the code base should get me on the right path as I may be way off by the sounds of it, even though it seems to work very well for what I need in 1.11.
>>>>>
>>>>> thanks
>>>>>
>>>>>
>>>>> On 04/21/2017 09:32 PM, Julian Hyde wrote:
>>>>>> MockCatalogReader was created for testing, but we don’t intend people to use it (or a modified version of it) in projects that use Calcite.
>>>>>>
>>>>>> CalciteSchema is a private class (the Java class is marked “public” only because it is used across several packages) and we don’t intend people to use it.
>>>>>>
>>>>>> Our intended extension point is Schema. Write your own instance of that API to go and get table definitions on demand. You need to write a getTableNames() method that returns all table names as a set, but you can create the table definitions one at a time when getTable(String name) is called.
>>>>>>
>>>>>> In https://issues.apache.org/jira/browse/CALCITE-1742 <https://issues.apache.org/jira/browse/CALCITE-1742> we are discussing possibly changing this model, but hopefully in a compatible way.
>>>>>>
>>>>>> Julian
>>>>>>
>>>>>>
>>>>>>> On Apr 21, 2017, at 8:19 PM, LogSplitter <lo...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am just in the process of updating to the latest version of calcite for my project and have run into an issue and not sure whether I am missing something.
>>>>>>>
>>>>>>> I currently use a process based on some of the code around MockCatalogReader to create a catalog and allow my sql to get parsed to relational algebra, which is all I use from the calcite side of things.
>>>>>>>
>>>>>>> I was able to dynamically look up metadata previously in the code by having the CatalogReader getTable code lazily load the metadata items as it received request for tables.
>>>>>>>
>>>>>>> It seems with the 1.12 code and the requirement to use calciteSchema it expects all the schemas and tables to be available all the time.
>>>>>>>
>>>>>>> I am wondering if my lazy load approach is a mistake or if I should work on the calciteSchema interface to allow it to continue with this behaviour.
>>>>>>>
>>>>>>> Does anyone have thoughts on if what I am trying to do makes sense.
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>>
>
Re: Question about CalciteSchema in 1.12
Posted by LogSplitter <lo...@gmail.com>.
Julian,
Thanks for your reply.
I muddled through and built a planner based on the exact code as Planner
for now with the config I needed.
I did notice someone else had same issues and has built a PR with
ability to pass a config, here
https://issues.apache.org/jira/browse/CALCITE-1849, once this one lands
we should be good.
I like the system field config idea, I will try getting that working.
Thanks
On 07/04/2017 06:17 PM, Julian Hyde wrote:
> Planner doesn’t support what you need right now. Can you log a JIRA case please?
>
> You don’t need to sub-class SqlToRelConverter to control its behavior, just pass in a SqlToRelConverter.Config that says exactly what its behavior should be. I see that PlannerImpl.rel builds a config but doesn’t allow it to be customized.
>
> Also change SqlToRelConverter.getSystemFields() so that it gets system fields from a member variable, populated from Config:
>
> public interface Config {
> Function<RelDataTypeFactory, List<RelDataTypeField>> systemFieldsFactory();
> }
>
> SqlToRelConverter() {
> this.systemFieldList =
> ImmutableList.copyOf(config.systemFieldsFactory().apply(typeFactory));
> }
>
> Julian
>
>
>> On Jul 4, 2017, at 12:03 AM, LogSplitter <lo...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have been trying to get planner to work and it is really close but I seem to get into issues due to one special requirement I have where I have to remove a system column on 'select *' statements.
>>
>> Background, we used to remove the special columns by processing through the SqlNodeLIst of the validated query and rebuild the select list (setSelectList()) with the system column I wanted removed. This approach doesn't seem to work in the planner world as I believe I need to do a validate after the selectList is modified but the planner model doesn't like it when you don't do all steps exactly as it expected (these types of errors Exception occurred: cannot move to STATE_3_PARSED from STATE_4_VALIDATED)
>>
>> I asked in a different thread how to handle the system columns and was told to try making my own SqlToRelConverter.getSystemFields() but I could not see how I could plug my own SqlToRelConverter into the planner model.
>>
>> So currently I am scratching my head stuck on an old version of calcite wonder which direction I should head. Ideally I would like to use the Planner model but need a little pointer of how I get a custom SqlToRelConverter.getSystemFields() hooked up.
>>
>> thanks
>>
>>
>> On 04/22/2017 10:07 AM, Julian Hyde wrote:
>>> Have you tried org.apache.calcite.tools.Planner? There are examples in org.apache.calcite.tools.PlannerTest. Barely a mention of SqlValidator, let alone CalciteSchema.
>>>
>>>
>>>> On Apr 21, 2017, at 10:17 PM, LogSplitter <lo...@gmail.com> wrote:
>>>>
>>>> Julian,
>>>>
>>>> So I am thinking that I am probably missing something here as I seem to need a CatalogReader to be able to get a SqlValidatorImpl to enable me to call validate to get the relational algebra. What would be the recommended way? Pointing me to a sample in the code base should get me on the right path as I may be way off by the sounds of it, even though it seems to work very well for what I need in 1.11.
>>>>
>>>> thanks
>>>>
>>>>
>>>> On 04/21/2017 09:32 PM, Julian Hyde wrote:
>>>>> MockCatalogReader was created for testing, but we don’t intend people to use it (or a modified version of it) in projects that use Calcite.
>>>>>
>>>>> CalciteSchema is a private class (the Java class is marked “public” only because it is used across several packages) and we don’t intend people to use it.
>>>>>
>>>>> Our intended extension point is Schema. Write your own instance of that API to go and get table definitions on demand. You need to write a getTableNames() method that returns all table names as a set, but you can create the table definitions one at a time when getTable(String name) is called.
>>>>>
>>>>> In https://issues.apache.org/jira/browse/CALCITE-1742 <https://issues.apache.org/jira/browse/CALCITE-1742> we are discussing possibly changing this model, but hopefully in a compatible way.
>>>>>
>>>>> Julian
>>>>>
>>>>>
>>>>>> On Apr 21, 2017, at 8:19 PM, LogSplitter <lo...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am just in the process of updating to the latest version of calcite for my project and have run into an issue and not sure whether I am missing something.
>>>>>>
>>>>>> I currently use a process based on some of the code around MockCatalogReader to create a catalog and allow my sql to get parsed to relational algebra, which is all I use from the calcite side of things.
>>>>>>
>>>>>> I was able to dynamically look up metadata previously in the code by having the CatalogReader getTable code lazily load the metadata items as it received request for tables.
>>>>>>
>>>>>> It seems with the 1.12 code and the requirement to use calciteSchema it expects all the schemas and tables to be available all the time.
>>>>>>
>>>>>> I am wondering if my lazy load approach is a mistake or if I should work on the calciteSchema interface to allow it to continue with this behaviour.
>>>>>>
>>>>>> Does anyone have thoughts on if what I am trying to do makes sense.
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>>
Re: Question about CalciteSchema in 1.12
Posted by Julian Hyde <jh...@apache.org>.
Planner doesn’t support what you need right now. Can you log a JIRA case please?
You don’t need to sub-class SqlToRelConverter to control its behavior, just pass in a SqlToRelConverter.Config that says exactly what its behavior should be. I see that PlannerImpl.rel builds a config but doesn’t allow it to be customized.
Also change SqlToRelConverter.getSystemFields() so that it gets system fields from a member variable, populated from Config:
public interface Config {
Function<RelDataTypeFactory, List<RelDataTypeField>> systemFieldsFactory();
}
SqlToRelConverter() {
this.systemFieldList =
ImmutableList.copyOf(config.systemFieldsFactory().apply(typeFactory));
}
Julian
> On Jul 4, 2017, at 12:03 AM, LogSplitter <lo...@gmail.com> wrote:
>
> Hi,
>
> I have been trying to get planner to work and it is really close but I seem to get into issues due to one special requirement I have where I have to remove a system column on 'select *' statements.
>
> Background, we used to remove the special columns by processing through the SqlNodeLIst of the validated query and rebuild the select list (setSelectList()) with the system column I wanted removed. This approach doesn't seem to work in the planner world as I believe I need to do a validate after the selectList is modified but the planner model doesn't like it when you don't do all steps exactly as it expected (these types of errors Exception occurred: cannot move to STATE_3_PARSED from STATE_4_VALIDATED)
>
> I asked in a different thread how to handle the system columns and was told to try making my own SqlToRelConverter.getSystemFields() but I could not see how I could plug my own SqlToRelConverter into the planner model.
>
> So currently I am scratching my head stuck on an old version of calcite wonder which direction I should head. Ideally I would like to use the Planner model but need a little pointer of how I get a custom SqlToRelConverter.getSystemFields() hooked up.
>
> thanks
>
>
> On 04/22/2017 10:07 AM, Julian Hyde wrote:
>> Have you tried org.apache.calcite.tools.Planner? There are examples in org.apache.calcite.tools.PlannerTest. Barely a mention of SqlValidator, let alone CalciteSchema.
>>
>>
>>> On Apr 21, 2017, at 10:17 PM, LogSplitter <lo...@gmail.com> wrote:
>>>
>>> Julian,
>>>
>>> So I am thinking that I am probably missing something here as I seem to need a CatalogReader to be able to get a SqlValidatorImpl to enable me to call validate to get the relational algebra. What would be the recommended way? Pointing me to a sample in the code base should get me on the right path as I may be way off by the sounds of it, even though it seems to work very well for what I need in 1.11.
>>>
>>> thanks
>>>
>>>
>>> On 04/21/2017 09:32 PM, Julian Hyde wrote:
>>>> MockCatalogReader was created for testing, but we don’t intend people to use it (or a modified version of it) in projects that use Calcite.
>>>>
>>>> CalciteSchema is a private class (the Java class is marked “public” only because it is used across several packages) and we don’t intend people to use it.
>>>>
>>>> Our intended extension point is Schema. Write your own instance of that API to go and get table definitions on demand. You need to write a getTableNames() method that returns all table names as a set, but you can create the table definitions one at a time when getTable(String name) is called.
>>>>
>>>> In https://issues.apache.org/jira/browse/CALCITE-1742 <https://issues.apache.org/jira/browse/CALCITE-1742> we are discussing possibly changing this model, but hopefully in a compatible way.
>>>>
>>>> Julian
>>>>
>>>>
>>>>> On Apr 21, 2017, at 8:19 PM, LogSplitter <lo...@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am just in the process of updating to the latest version of calcite for my project and have run into an issue and not sure whether I am missing something.
>>>>>
>>>>> I currently use a process based on some of the code around MockCatalogReader to create a catalog and allow my sql to get parsed to relational algebra, which is all I use from the calcite side of things.
>>>>>
>>>>> I was able to dynamically look up metadata previously in the code by having the CatalogReader getTable code lazily load the metadata items as it received request for tables.
>>>>>
>>>>> It seems with the 1.12 code and the requirement to use calciteSchema it expects all the schemas and tables to be available all the time.
>>>>>
>>>>> I am wondering if my lazy load approach is a mistake or if I should work on the calciteSchema interface to allow it to continue with this behaviour.
>>>>>
>>>>> Does anyone have thoughts on if what I am trying to do makes sense.
>>>>>
>>>>> thanks
>>>>>
>>>>>
>