You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ShiXing (Created) (JIRA)" <ji...@apache.org> on 2012/04/05 12:07:25 UTC

[jira] [Created] (HBASE-5723) Simple Design of Secondary Index

Simple Design of Secondary Index
--------------------------------

                 Key: HBASE-5723
                 URL: https://issues.apache.org/jira/browse/HBASE-5723
             Project: HBase
          Issue Type: New Feature
          Components: coprocessors
            Reporter: ShiXing
            Priority: Minor
         Attachments: Simple Design of HBase SecondaryIndex.pdf

Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249907#comment-13249907 ] 

Zhihong Yu commented on HBASE-5723:
-----------------------------------

@Todd:
Do you think the attached proposal belongs to HBASE-3340 ?
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "ShiXing (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247127#comment-13247127 ] 

ShiXing commented on HBASE-5723:
--------------------------------

Difficulty and risk
1. During primary table compact, the StoreScanner just read
family.getMaxVersions() versions' data, the old data will be skiped, but the
old data's indices should be deleted, So we must read all version data for
indices' delete, and family.getMaxVersions() version for store's compaction.

Also the timestamp of columns that we want to delete the indices should also have not TimeRange, now it is System.CurrentTimeMillis() - ttl;
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "Xing Shi (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248021#comment-13248021 ] 

Xing Shi commented on HBASE-5723:
---------------------------------

Todd, 
    you mean the consistency of index table? Because the index table is also a HBase table, so it will use the mvcc to ensure the consistency of the index table. 

Mainly there are 2 steps when do put.
Step1 : It will insert data into index table.
Step2 : It will insert data into primary table.

After step1, the client can read the index, but may be the step2 has not done. So client will find there is no data are associated with the index, and it will read the old version index. Here it is weak consistency, and at last the index and data will be eventual consistency except as follows:.

If the step2 failed, we will not delete the new index, the index is not correct(there is no row in the primary table associated with this index by the rowkey, the value, and the same timestamp) ant it is Weak consistency.

For these stale indices, we will not delete it forever(may be we can delete them by period job to detect the not correct indice).
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "Jesse Yates (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248426#comment-13248426 ] 

Jesse Yates commented on HBASE-5723:
------------------------------------

There is a lot of work that the Lily guys have done along the lines of ensuring consistency, but basically revolves around using a WAL for making sure the edits get back in. Personally, I find their approach a bit heavy handed (esp with the coprocessor framework _now_ available, though it makes sense before CPs). They also have done a lot of work (open source!) for indexing different data types, integration with Solr, etc. Either way, worth taking a dive into their stuff, if only for inspiration and seeing what people who have thought a lot about this problem have some up with.
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-5723) Simple Design of Secondary Index

Posted by "ShiXing (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ShiXing updated HBASE-5723:
---------------------------

    Attachment: Simple Design of HBase SecondaryIndex.pdf
    
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248024#comment-13248024 ] 

Todd Lipcon commented on HBASE-5723:
------------------------------------

If this implementation isn't giving you any consistency, then why is it any better than just having the client do two puts?

I think for us to consider adding an indexing library to HBase proper, it needs to have stronger semantics. The idea that an index may be incorrect in the case of failures does not seem like something we should ship.
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247438#comment-13247438 ] 

Todd Lipcon commented on HBASE-5723:
------------------------------------

The design doc doesn't make any reference to consistency. How do you ensure consistency of the index? To do proper indexes you really need cross-table transactions, or at least a mechanism for eventual consistency if that's all you plan on providing.
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index

Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249927#comment-13249927 ] 

Todd Lipcon commented on HBASE-5723:
------------------------------------

This seems to give even fewer guarantees than HBASE-3340 (which as I understood it used the WAL to ensure reasonably timely consistency of the index table). I'm still of the opinion that these things should start as separate projects, rather than inside HBase core, and only if we see a high demand for them and good maintenance, we should pull it in. I don't think there is much demand for non-consistent indexes.
                
> Simple Design of Secondary Index
> --------------------------------
>
>                 Key: HBASE-5723
>                 URL: https://issues.apache.org/jira/browse/HBASE-5723
>             Project: HBase
>          Issue Type: New Feature
>          Components: coprocessors
>            Reporter: ShiXing
>            Priority: Minor
>         Attachments: Simple Design of HBase SecondaryIndex.pdf
>
>
> Use coprocessor to create index. And primary tables' compaction to purge the stale data. 
> Attach file is the Design of the Seconday Index.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira