Apache Calcite Enumerator Optimization

["Leur, Maxime" <Ma...@fisglobal.com>]

Hi all,

First thanks for your project, it is very great and help me a lot to provide SQL interface to own internal DB.
I succeed to implement my own Schema/Table with the tutorial but I have performance issues on high volume.

I mean when I have 1 millions of rows with many columns (> 20), it takes too long time to execute request.
I check the code and I think that is due to the fact that I must convert my internal object (row) to "Object[]" (org.apache.calcite.schema.FilterableTable).

It is possible to give a Map / List or any proxy Object to "Apache Calcite" instead of "Object[]" array?
I check some Table implementation on GitHub (MongoDB, Cassandra) and debug code in Eclipse but I do not succeed.
I think that is linked to "org.apache.calcite.adapter.java.AbstractQueryableTable" but despite all my efforts I cannot understand how to do it.

Could you please help me on that subject?

Regards,

Maxime Leur
J2EE Architect
Global Trading / TCC
T:  +33 (0)1 53 40 57 46
C:  1746
E: maxime.leur@fisglobal.com<ma...@fisglobal.com>
FIS | Empowering the Financial World [cid:image005.png@01D11265.2C7F1D60] <https://www.facebook.com/FIStoday> [cid:image006.png@01D11265.2C7F1D60] <https://twitter.com/FISGlobal> [cid:image007.png@01D11265.2C7F1D60] <https://www.linkedin.com/company/fis>

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Apache Calcite Enumerator Optimization

[Julian Hyde <jh...@apache.org>]

It’s definitely possible, but it would be a fair amount of work on your part. The class JavaRowFormat already describes how rows are represented within generated code, and it already has a value LIST.

Now, FilterableTable is slightly different. It has to produce rows in Object[] format. Why? Because it (and ScannableTable and ProjectableFilterableTable) is intentionally a simple API.

If you have more complex requirements, you should use QueryableTable. See EnumerableTableScan.deduceElementType():

  public static Class deduceElementType(Table table) {
    if (table instanceof QueryableTable) {
      final QueryableTable queryableTable = (QueryableTable) table;
      final Type type = queryableTable.getElementType();
      if (type instanceof Class) {
        return (Class) type;
      } else {
        return Object[].class;
      }
    } else if (table instanceof ScannableTable
        || table instanceof FilterableTable
        || table instanceof ProjectableFilterableTable
        || table instanceof StreamableTable) {
      return Object[].class;
    } else {
      return Object.class;
    }
  }

Then you will need to decide whether rows remain as List all the way through the pipeline or whether you convert them to Object[] at some point.

Julian


> On Jan 13, 2017, at 1:38 AM, Leur, Maxime <Ma...@fisglobal.com> wrote:
> 
> Hi all,
>  
> First thanks for your project, it is very great and help me a lot to provide SQL interface to own internal DB.
> I succeed to implement my own Schema/Table with the tutorial but I have performance issues on high volume.
>  
> I mean when I have 1 millions of rows with many columns (> 20), it takes too long time to execute request.
> I check the code and I think that is due to the fact that I must convert my internal object (row) to “Object[]” (org.apache.calcite.schema.FilterableTable).
>  
> It is possible to give a Map / List or any proxy Object to “Apache Calcite” instead of “Object[]” array?
> I check some Table implementation on GitHub (MongoDB, Cassandra) and debug code in Eclipse but I do not succeed.
> I think that is linked to “org.apache.calcite.adapter.java.AbstractQueryableTable” but despite all my efforts I cannot understand how to do it.
>  
> Could you please help me on that subject?
>  
> Regards,
>  
> Maxime Leur
> J2EE Architect
> Global Trading / TCC
> T:  +33 (0)1 53 40 57 46
> C:  1746
> E: maxime.leur@fisglobal.com <ma...@fisglobal.com>
> FIS | Empowering the Financial World  <https://www.facebook.com/FIStoday> <https://twitter.com/FISGlobal> <https://www.linkedin.com/company/fis>
>  
> _____________
> The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.