You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by "Keith Turner (Created) (JIRA)" <ji...@apache.org> on 2011/12/01 22:18:40 UTC
[jira] [Created] (GORA-65) Implement Accumulo datastore for Gora
Implement Accumulo datastore for Gora
-------------------------------------
Key: GORA-65
URL: https://issues.apache.org/jira/browse/GORA-65
Project: Gora
Issue Type: New Feature
Reporter: Keith Turner
Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (GORA-65) Implement Accumulo
datastore for Gora
Posted by "Keith Turner (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172673#comment-13172673 ]
Keith Turner edited comment on GORA-65 at 12/19/11 10:22 PM:
-------------------------------------------------------------
GORA-65-1.patch is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
was (Author: kturner):
[#GORA-65-1.patch] is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173607#comment-13173607 ]
Keith Turner commented on GORA-65:
----------------------------------
Oh one other major difference is paritioning a query. I will explain this by example. Suppose I have an gora-accumulo table where the key is a long. Suppose this table has the following splits. There are two reasons these split points are not 8 bytes. Users can insert split points. Also, when Accumulo chooses a split point, it will choose the shortest possible string. So split point do not always come from rows in the table.
{noformat}
0x20
0x60
{noformat}
When partitioning a query over the entire table, I have to take the 2 splits above and generate three partition queries that cover the entire table but do not overlap. Also the queries need to align exactly with accumulo tablets (like HBase regions) to maintain locality. The fact that the split points are not 8 byte also makes it tricky, they will not decode as a long. So I generate three gora queries with the following start and end keys. These queries cover all possible longs in the table. The queries map to accumulo tablets which have exclusive start and inclusive end keys.
{noformat}
query 1 start key=null end key=0x1fffffffffffffff
query 2 start key=0x2000000000000000 end key=0x5fffffffffffffff
query 3 start key=0x6000000000000000 end key=null
{noformat}
When the key is not fixed width (like a long or int) and its a String, then its much easier to create the partition queries.
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173564#comment-13173564 ]
Lewis John McGibbney commented on GORA-65:
------------------------------------------
Another thing, it would be great to get the MR tests implemented but as Gora doesn't rely upon Hadoop this is not essential. In addition, unfortunately to date we've not managed to implement gora-cassandra tests yet so this is another reason to further support your patch and its backing for integration into the codebase. Thanks
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173581#comment-13173581 ]
Keith Turner commented on GORA-65:
----------------------------------
We just released 1.3.5 yesterday, we have not yet put jars in a public maven repo. We are still working on that. I can make the patch use 1.3.5. Then the patch would build out of the box once we get the jars in a public repo.
We plan to start working on releasing 1.4 now. The patch leverages some newer features. Depending on when we release 1.4 and when the next Gora release, I would probably like to base it against 1.4 if possible. We have just started working on 1.5, so that release will not be soon. Upon entering incubation, 1.3 was our stable branch and we were just about to release 1.4.
The code should be pretty similar to the HBase data store. One place were I differed is that I made it possible for the user to choose different ways of encoding the data in Accumulo. The BinaryEncode is similar to what HBase does. The SignedBinaryEncoder encodes the data such that negative numbers sort first lexographically. This is useful if the Gora key is long and you want a query from -5 to 5 to work. If you want to use long as the key, but do not want to treat it as an unsigned long then you would not want to use this encoding. The HexEncoder uses ascii hex to encode long, int, float, double, I like this for debugging.
The mapping file is similar to hbase except for table setup. The table part of a mapping supports setting per table config key value pairs. This is how all tables are configured in Accumulo. Using this mechanism cache, localitly groups, bloom filters, iterators, compression, etc can all be configured for a table by setting key value pairs. The class part of the mapping is the same, except I added the encoder config.
Is this the type of high level info you were looking for?
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161181#comment-13161181 ]
Keith Turner commented on GORA-65:
----------------------------------
I am currently working on implementing this and plan to post a patch when I have some basic functionality that passes the basic test suite.
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207699#comment-13207699 ]
Lewis John McGibbney commented on GORA-65:
------------------------------------------
I think we will be ready to release 0.2 when the majority of the issues [1] are addressed. I would like to think this would be achievable within 1st qtr 2012. Strictly speaking, none are bugs, however the website documentation and sqlbuilder issues are blockers in my opinion.
[1] https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+GORA+AND+resolution+%3D+Unresolved+AND+fixVersion+%3D+%220.2%22+ORDER+BY+priority+DESC&mode=hide
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Apache Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206322#comment-13206322 ]
Lewis John McGibbney commented on GORA-65:
------------------------------------------
Hi Keith, what problems were you having with implementing the GoraAccumuloTestDriver to run the existing tests from gora-core/../../../DataStoreTestUtil against? Also to pick you up on something
{bq}
We just released 1.3.5 yesterday, we have not yet put jars in a public maven repo. We are still working on that. I can make the patch use 1.3.5. Then the patch would build out of the box once we get the jars in a public repo.
{bq}
I think we should support 1.3.5 when we make our 0.2 release then simply shift to support 1.4.0-incubating-SNAPSHOT in gora trunk.
I would be happy to see this going in to Gora ASAP, based upon the TestDriver situation. I really like what you have done with the encoding, thanks for being verbose about it above.
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Apache Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207707#comment-13207707 ]
Keith Turner commented on GORA-65:
----------------------------------
Documentation is something I need to do for gora-accumulo
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Apache Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated GORA-65:
-----------------------------
Attachment: GORA-65-1.patch
GORA-65-1.patch is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (GORA-65) Implement Accumulo
datastore for Gora
Posted by "Keith Turner (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172673#comment-13172673 ]
Keith Turner edited comment on GORA-65 at 12/19/11 10:17 PM:
-------------------------------------------------------------
[^GORA-65-1.patch] is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
was (Author: kturner):
GORA-65-1.patch is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173562#comment-13173562 ]
Lewis John McGibbney commented on GORA-65:
------------------------------------------
Hi Keith. As this is obviously a meaty patch, I'm just trying to take it bit by bit the now. I'm assuming that you's guys @ Accumulo are about to release Accumulo 1.4.X or are you's about to release 1.5.X? I had a look @ 1.3.5 but haven't gone any higher/newer. It suppose on the face of it, it doesn't really make much of a difference (unless there are major API changes), but I would like to get my hands on a similar version of Accumulo that you wrote the gora-module against to try it out. It looks great.
I wonder if you could provide a bit of detail on the structure of the module and the tests for us guys who might not have immediate gora-accumulo use cases but that would really like to test drive the code before commenting. If we can get this integrated into Gora, then there is scope for further implement it within the Nutchgora branch (under development)... which again would be excellent. Thanks for any info.
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (GORA-65) Implement Accumulo
datastore for Gora
Posted by "Keith Turner (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172673#comment-13172673 ]
Keith Turner edited comment on GORA-65 at 12/19/11 10:18 PM:
-------------------------------------------------------------
[#GORA-65-1.patch] is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
was (Author: kturner):
[^GORA-65-1.patch] is a work in progress, posting what I have done so far for review. The issues that I know of with this patch are :
* depends on unreleased version of accumulo
* does not run map reduce unit test
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207697#comment-13207697 ]
Keith Turner commented on GORA-65:
----------------------------------
The problems were with Mock Accumulo. This is a simple implementation of Accumulo that allows unit test to run in process very quickly. It was first introduced in 1.3 and is more mature in 1.4. I ran into some problems getting the test to work against 1.3 mock Accumulo. The test work fine against 1.4. Whats the timeline for 0.2? I do not want to hold it up. I am still very busy w/ testing 1.4 and fixing bugs inorder to get it ready for release.
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Apache Gora
> Issue Type: New Feature
> Reporter: Keith Turner
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GORA-65) Implement Accumulo datastore for Gora
Posted by "Lewis John McGibbney (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney updated GORA-65:
-------------------------------------
Affects Version/s: 0.2
Fix Version/s: 0.3
> Implement Accumulo datastore for Gora
> -------------------------------------
>
> Key: GORA-65
> URL: https://issues.apache.org/jira/browse/GORA-65
> Project: Apache Gora
> Issue Type: New Feature
> Affects Versions: 0.2
> Reporter: Keith Turner
> Fix For: 0.3
>
> Attachments: GORA-65-1.patch
>
>
> Accumulo is a BigTable implementation similar to HBase. Accumulo would be a great fit for another Gora datastore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira