You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by Indhumathi M <in...@gmail.com> on 2020/02/05 12:03:18 UTC
[Discussion] Support Secondary Index on Carbon Table
Hi Community,
Currently we have datamaps like,* default datamaps* which are block and
blocklet and *coarse grained datamaps* like bloom, and *fine grained
datamaps* like lucene
which helps in better pruning during query. What if we introduce another
kind of datamap which can hold blockletId as index? Initial level, we call
it as index which
will work as a child table to the main table like we have MV in our current
code.
Yes, lets introduce the secondary index to carbon table which will be the
child table to main table and it can be created on column like we create
lucene datamap,
where we give index columns to create index. In a similar way, we create
secondary index on column, so indexes on these column will be blocklet IDs
which will
help in better pruning and faster query when we have a filter query on the
index column.
Currenlty we will take it as index table and then later part we will make
it inline to datamap interface.
So design document is attached in JIRA, please give your suggestion/inputs.
JIRA Link: CARBONDATA-3680
<https://issues.apache.org/jira/browse/CARBONDATA-3680>
Thanks & Regards,
Indhumathi M
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by niuge <37...@qq.com>.
+1
I have a suggestion.
Comparatively, query hint will be better to avoid queries push downed to SI
table.
Example:
SELECT /*disable_si*/ * FROM main_table WHERE name='abc'
Regards,
Zhi Liu
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by Kumar Vishal <ku...@gmail.com>.
+1
-Regards
Kumar Vishal
On Wed, 5 Feb 2020 at 8:08 PM, Ajantha Bhat <aj...@gmail.com> wrote:
> +1
>
> On Wed, 5 Feb, 2020, 8:02 pm Ravindra Pesala, <ra...@gmail.com>
> wrote:
>
> > +1
> >
> > Regards,
> > Ravindra.
> >
> > On Wed, 5 Feb 2020 at 8:03 PM, Indhumathi M <in...@gmail.com>
> > wrote:
> >
> > > Hi Community,
> > >
> > > Currently we have datamaps like,* default datamaps* which are block and
> > > blocklet and *coarse grained datamaps* like bloom, and *fine grained
> > > datamaps* like lucene
> > > which helps in better pruning during query. What if we introduce
> another
> > > kind of datamap which can hold blockletId as index? Initial level, we
> > call
> > > it as index which
> > > will work as a child table to the main table like we have MV in our
> > current
> > > code.
> > >
> > > Yes, lets introduce the secondary index to carbon table which will be
> the
> > > child table to main table and it can be created on column like we
> create
> > > lucene datamap,
> > > where we give index columns to create index. In a similar way, we
> create
> > > secondary index on column, so indexes on these column will be blocklet
> > IDs
> > > which will
> > > help in better pruning and faster query when we have a filter query on
> > the
> > > index column.
> > >
> > > Currenlty we will take it as index table and then later part we will
> make
> > > it inline to datamap interface.
> > >
> > > So design document is attached in JIRA, please give your
> > suggestion/inputs.
> > >
> > > JIRA Link: CARBONDATA-3680
> > > <https://issues.apache.org/jira/browse/CARBONDATA-3680>
> > >
> > > Thanks & Regards,
> > > Indhumathi M
> > >
> > --
> > Thanks & Regards,
> > Ravi
> >
>
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by Ajantha Bhat <aj...@gmail.com>.
+1
On Wed, 5 Feb, 2020, 8:02 pm Ravindra Pesala, <ra...@gmail.com> wrote:
> +1
>
> Regards,
> Ravindra.
>
> On Wed, 5 Feb 2020 at 8:03 PM, Indhumathi M <in...@gmail.com>
> wrote:
>
> > Hi Community,
> >
> > Currently we have datamaps like,* default datamaps* which are block and
> > blocklet and *coarse grained datamaps* like bloom, and *fine grained
> > datamaps* like lucene
> > which helps in better pruning during query. What if we introduce another
> > kind of datamap which can hold blockletId as index? Initial level, we
> call
> > it as index which
> > will work as a child table to the main table like we have MV in our
> current
> > code.
> >
> > Yes, lets introduce the secondary index to carbon table which will be the
> > child table to main table and it can be created on column like we create
> > lucene datamap,
> > where we give index columns to create index. In a similar way, we create
> > secondary index on column, so indexes on these column will be blocklet
> IDs
> > which will
> > help in better pruning and faster query when we have a filter query on
> the
> > index column.
> >
> > Currenlty we will take it as index table and then later part we will make
> > it inline to datamap interface.
> >
> > So design document is attached in JIRA, please give your
> suggestion/inputs.
> >
> > JIRA Link: CARBONDATA-3680
> > <https://issues.apache.org/jira/browse/CARBONDATA-3680>
> >
> > Thanks & Regards,
> > Indhumathi M
> >
> --
> Thanks & Regards,
> Ravi
>
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by Ravindra Pesala <ra...@gmail.com>.
+1
Regards,
Ravindra.
On Wed, 5 Feb 2020 at 8:03 PM, Indhumathi M <in...@gmail.com> wrote:
> Hi Community,
>
> Currently we have datamaps like,* default datamaps* which are block and
> blocklet and *coarse grained datamaps* like bloom, and *fine grained
> datamaps* like lucene
> which helps in better pruning during query. What if we introduce another
> kind of datamap which can hold blockletId as index? Initial level, we call
> it as index which
> will work as a child table to the main table like we have MV in our current
> code.
>
> Yes, lets introduce the secondary index to carbon table which will be the
> child table to main table and it can be created on column like we create
> lucene datamap,
> where we give index columns to create index. In a similar way, we create
> secondary index on column, so indexes on these column will be blocklet IDs
> which will
> help in better pruning and faster query when we have a filter query on the
> index column.
>
> Currenlty we will take it as index table and then later part we will make
> it inline to datamap interface.
>
> So design document is attached in JIRA, please give your suggestion/inputs.
>
> JIRA Link: CARBONDATA-3680
> <https://issues.apache.org/jira/browse/CARBONDATA-3680>
>
> Thanks & Regards,
> Indhumathi M
>
--
Thanks & Regards,
Ravi
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by manish gupta <to...@gmail.com>.
+1
Regards
Manish Gupta
On Thu, 6 Feb 2020 at 1:50 PM, David CaiQiang <da...@gmail.com> wrote:
> +1
>
>
>
> -----
> Best Regards
> David Cai
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by David CaiQiang <da...@gmail.com>.
+1
-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
回复： [Discussion] Support Secondary Index on Carbon Table
Posted by Jacky Li <ja...@qq.com>.
+1
Thanks for proposing this :)
Regards,
Jacky
------------------ 原始邮件 ------------------
发件人: "Kunal Kapoor"<kunalkapoor642@gmail.com>;
发送时间: 2020年2月6日(星期四) 凌晨2:13
收件人: "dev"<dev@carbondata.apache.org>;
主题: Re: [Discussion] Support Secondary Index on Carbon Table
+1
Thanks
Kunal Kapoor
On Wed, Feb 5, 2020, 5:33 PM Indhumathi M <indhumathim27@gmail.com> wrote:
> Hi Community,
>
> Currently we have datamaps like,* default datamaps* which are block and
> blocklet and *coarse grained datamaps* like bloom, and *fine grained
> datamaps* like lucene
> which helps in better pruning during query. What if we introduce another
> kind of datamap which can hold blockletId as index? Initial level, we call
> it as index which
> will work as a child table to the main table like we have MV in our current
> code.
>
> Yes, lets introduce the secondary index to carbon table which will be the
> child table to main table and it can be created on column like we create
> lucene datamap,
> where we give index columns to create index. In a similar way, we create
> secondary index on column, so indexes on these column will be blocklet IDs
> which will
> help in better pruning and faster query when we have a filter query on the
> index column.
>
> Currenlty we will take it as index table and then later part we will make
> it inline to datamap interface.
>
> So design document is attached in JIRA, please give your suggestion/inputs.
>
> JIRA Link: CARBONDATA-3680
> <https://issues.apache.org/jira/browse/CARBONDATA-3680>
>
> Thanks & Regards,
> Indhumathi M
>
Re: [Discussion] Support Secondary Index on Carbon Table
Posted by Kunal Kapoor <ku...@gmail.com>.
+1
Thanks
Kunal Kapoor
On Wed, Feb 5, 2020, 5:33 PM Indhumathi M <in...@gmail.com> wrote:
> Hi Community,
>
> Currently we have datamaps like,* default datamaps* which are block and
> blocklet and *coarse grained datamaps* like bloom, and *fine grained
> datamaps* like lucene
> which helps in better pruning during query. What if we introduce another
> kind of datamap which can hold blockletId as index? Initial level, we call
> it as index which
> will work as a child table to the main table like we have MV in our current
> code.
>
> Yes, lets introduce the secondary index to carbon table which will be the
> child table to main table and it can be created on column like we create
> lucene datamap,
> where we give index columns to create index. In a similar way, we create
> secondary index on column, so indexes on these column will be blocklet IDs
> which will
> help in better pruning and faster query when we have a filter query on the
> index column.
>
> Currenlty we will take it as index table and then later part we will make
> it inline to datamap interface.
>
> So design document is attached in JIRA, please give your suggestion/inputs.
>
> JIRA Link: CARBONDATA-3680
> <https://issues.apache.org/jira/browse/CARBONDATA-3680>
>
> Thanks & Regards,
> Indhumathi M
>