You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Erik Holstad (JIRA)" <ji...@apache.org> on 2009/04/01 19:06:13 UTC
[jira] Created: (HBASE-1304) New client server implementation of
how gets and puts are handled.
New client server implementation of how gets and puts are handled.
-------------------------------------------------------------------
Key: HBASE-1304
URL: https://issues.apache.org/jira/browse/HBASE-1304
Project: Hadoop HBase
Issue Type: Improvement
Affects Versions: 0.20.0
Reporter: Erik Holstad
Assignee: Erik Holstad
Priority: Blocker
Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-1304) New client server implementation
of how gets and puts are handled.
Posted by Ryan Rawson <ry...@gmail.com>.
The way the delete tracker works (for the delete case) takes a delete at
time t then the next put it encounters at time t-n it removes and then lifts
the delete filter. I think this makes sense, especially since for time "t" a
put sorts before a delete.
If a user wants to delete a value at time "j" then they should stamp the
delete at j+1. If we want to allow deletes set to time "t" to affect puts at
time t, we'll need to change the sorting of the type.
Neither of these will affect gets since the deletes affect older files only
will be maintained, furthermore the scanning minor compactor will ensure
this invariant.
On May 18, 2009 11:26 AM, "Jonathan Gray (JIRA)" <ji...@apache.org> wrote:
[
https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710436#action_12710436]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
I think everyone is in agreement about the future. Any stamp given on
anything (put or delete) that is > now, becomes now.
As far as a Delete at LATEST, yes right now it needs to be an exact match.
So, a Delete at LATEST will likely not do anything. Delete is for a single
version. DeleteColumn is for all versions <= a specified stamp. So if you
want to delete the latest version of something, you don't need to know it's
stamp as long as you don't mind clearing all versions before it. Any time
you want to delete _only_ a single version, you must know and specify the
exact stamp. I think that is fair and understandable, to do otherwise I
fear will sacrifice some of the good properties.
Perhaps we can figure a way to do a single-version delete of the latest
version of a column without affecting older ones. But it seems you'll have
to either actually perform some kind of Get during the delete to figure the
stamp, which there is not currently any of outside of touching Memcache, or
all of our DeleteTracker'ing will need to change how it handles Deletes.
Thoughts from others?
> New client server implementation of how gets and puts are handled. >
---------------------------...
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709883#action_12709883 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Stack
Are you mocking the test :)
I just think it is better to test the individual parts not a part of a cluster setup, and start up the cluster where you have a test that actually deals with sending data between the client and the server. Running all the unit test for HBase takes quite some time and I think that depends a little on that we are starting the clusters even though in some cases they are not needed to test the individual units, so just trying to keep testing time to a minimum, but that can of course be changed if you want to do it differently.
The print boolean is basically to not output any info when you have the test working, but will change to use the logger instead.
What is weird with the names?
The reason that the Scanner in the test takes a list of KeyValues is that at the time when the test was written there were no implementation of that scanner, so I had to create a private class to test the KeyValueHeap, so the way that inserting into the private test Scanner has really nothing to do with how it is going to look in the real scanner.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Attachment: HBASE-1304-fix.patch
fixes a test failure
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-fix.patch, HBASE-1304-ryan.patch, HBASE-1304-supp.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710267#action_12710267 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Ryan
The handling of the deletes for a scan is quite simple actually, much simpler than for a get query and like Jon said it might make sense to move everything to be a scan for correctness of the call, but we need to do timing tests on that first to see what would be the best deal. I don't really see how we not have to deal with deletes in both get/scan + compactions, unless we would do a major compaction every time, so there would never be any deletes in the system. Right now I think we would use the same logic for a compaction as for a scan call if everything goes as planned.
Are there anything wrong with the unit tests for the scan delete tracker? Or is there some other issue that makes you think it is broken? We haven't made a full client/server/client test yet, so I can't confirm that is works all the way, but please let me know what is broken.
@Stack
Yeah, I think that applying the deletes in memcache will keep it cleaner and we use less resources for the gets and the scans after that. I guess we don't have to use the fact that deletes only apply to earlier files, but I think it makes sense to use it to make gets faster, unless we make them scans and don't have to worry about it.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710385#action_12710385 ]
Andrew Purtell commented on HBASE-1304:
---------------------------------------
bq. once query matcher indicates we are 'done', we still have to continue consuming key values until the next row. We could flag this and skip this, but the 'scanner' would be useless for follow up queries.
What kind of performance implication is that? (Re: HBASE-867)
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707694#action_12707694 ]
stack commented on HBASE-1304:
------------------------------
@holstad
I'm fine using Bytes instead of ImmutableBytes. We just need to add the deprecated to ImmutableBytes.... Would we have to move Bytes to io package? Maybe best to just leave Bytes as utility class. Hadoop has BytesWritable already taken but its not what we want. Bytes should be instantiable so we can use it in things like Map<Bytes>. I like that bit of utility (though we should be careful where its used -- don't want to introduce new class instantiation when not needed and after doing all the work purging them).
Good stuff.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707549#action_12707549 ]
stack commented on HBASE-1304:
------------------------------
toString format needs to change in Delete, Get, etc., to match how we toString elsewhere using ruby-style for maps/arrays -- just for consistency sake. Ruby-style is pretty much how java-style works representing Maps and Lists and it matches shell.
Lets just drop RowLock from the new objects and use a long for lockid altogether? Thats all it is and one less object. Also io package not referencing item in client package.
Below is odd especially when Get does not implement Map
{code}
+ public Set<Map.Entry<byte[],TreeSet<byte[]>>> entrySet() {
+ return this.familyMap.entrySet();
+ }
{code}
Check for null array here:
{code}
+ public Result(KeyValue [] kvs) {
+ this.row = kvs[0].getRow();
+ this.kvs = kvs;
+ }
{code}
Does Result get instantiated server-side? Would be nice if it wasn't; if Clients made it, if the Server passed back List<KV>. Would save on Result instantiation and creation (and duplication) of a row in constructor (maybe getRow could be lazy.... don't create one unless its called).
Put some spaces in here so its more readable:
{code}
+ public SortedMap<byte[],SortedMap<byte[],SortedMap<Long,byte[]>>> getMap() {
{code}
Seems a pity creating a TreeMap when usual case is single family. Later we can optimize and only create the TreeMap on addition of second family?
Whats this:
{code}
+ /**
+ * Returns a {@link RowResult}.
+ * @return a RowResult
+ */
+ public RowResult rowResult() {
+ return new RowResult();
+ // return RowResult.createRowResult(kvs);
+ }
{code}
The below could be written as: "return this.kvs == null || this.kvs.length == 0;"
{code}
+ if(this.kvs == null || this.kvs.length == 0) {
+ return true;
+ }
+ return false;
{code}
Fix these: "+ for(int i=0; i<length; i++) " -- add spaces.
Lots of common code. Common base abstract class?
These should be static defines rather than created on each construction.
{code}
+ this.minStamp = Bytes.toBytes(0L);
+ this.maxStamp = Bytes.toBytes(Long.MAX_VALUE);
{code}
Regards TimeRange, are we for sure that most compares of timestamps are byte [] rather than long? This class is written for byte []. Surely compare of two longs is faster than creation of byte array representation and then byte compare?
These should be made on TimeRange creation? Or at least cached?
{code}
+ long min = Bytes.toLong(minStamp);
{code}
Above is minor. In general this TimeRange byte [] compare of longs seems like an optimization that'd never register its so small?
More to follow.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707691#action_12707691 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Stack
Thanks for all the comments.
About Bytes vs ImmutableBytesWritable, I totally forgot about Immutable, we can use that instead , the only thing is that the name is so long :) The place where this is used is when we building lists on the server from the KeyValues that we are looking at and don't want to copy. But it might be a good idea to keep Bytes as a clean util class, or not, but it doesn't really matter.
What we are doing now is that Jon is restructuring my code and I'm doing testing on it as we go, So to answer your question, no it doesn't work as it is right now, I'm still testing it. Puts are already done and are not that complicated, Deletes a little bit more complex but also done, but are maybe going to see rewrite. So what we have left to do is scanners and then compactions.
I think that we should have a version up and running at the end of next week, then we need to put the old client side methods back in there that uses the new api, but that should be too bad.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703768#action_12703768 ]
stack commented on HBASE-1304:
------------------------------
+ I think multifamily as an attribute a little redundant
+ MAX_FAMILY_LENGTH belongs better in KV
+ Ok on new names (was thinking we should start up a new hosting HTable class so you could keep method names like commit)
+ I think figuring the high-level class model, what the user sees, is probably the first order of business -- or at least belongs up there as a priority. The model needs to be tests for puts, deletes, gets and scans; everything.
+ Could GetTop be a Get against a family? Versions would in this case return that many of the family entries (the 'top' columns).
+ RowUpdates needs to passing KeyValues.
+ Regards Family object verses repeating family name in an array of arrays, lets say average get goes against one or two families, and that family name is generally short -- url, ts, page. Given that Family object with its internal List is how much? 50/80 bytes? Then on average I'd think current setup cheaper?
+ Family seems like a bad name for this class. Column would be better?
+ I didn't look at the isDeleted too deeply but seemed like duplication here with other parts of the patch. I know what you mean about carrying state -- can make it hard not duplicating code snippets -- but we should make an effort; when I see all this state-carrying code, I see lots of opportunity for bugs.... Needs good testing.
I'm not seeing a focus on zero-copy in this patch; passing KV out to client. Am I missing it or is it just not there yet?
I'm thinking that what with working out model, figuring out optimal client/server message passing, changing deletes to work in the new way, effectively rewriting regionserver again -- took me about a month to do it recently -- to implement the new server-side versions of new client-side objects, I'm thinking that 0.20.0 is 4-8 weeks out if we include this stuff. I think that too long to wait.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Attachments: hbase-1304-v1.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708476#action_12708476 ]
stack commented on HBASE-1304:
------------------------------
@jon
TreeSet is implementation of SortedSet interface. Was suggesting we use the Interface rather than the implementation. TreeSet also implements NavigableSet. NavigableSet is richer than SortedSet.
On this.comparator.getRawComparator, my fault. Was misreading. It looks right in code.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703706#action_12703706 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
+ Yes multiFamily is a boolean that I thought would live in HCD, but I don't have any preferences in this case and I don't know how the client would count that, but that might be really easy.
+ The reason that I put MAX_FMAILY_LENGTH in HConstant was that MAX_ROW_LENGTH was already in there so I thought that it was the place to put things like that.
+ In KeyValue, this might be the case that the stuff that I added are already in there, will try to change that when going through the code. Yes the set method is not needed, not sure right now why I put that there.
+ So the reasoning around the naming is that the names that I'm using now are just working names, I run both old implementations side by side, so I can compare the codes and behaviour between them. I don't intend to keep all the new names when I got this to work, only used for debugging purposes.
+ The reason behind having different classes for the different Get types is more a way to explain to the user that the calls behave differently and has really nothing to do with the underlying implementation, like you say they could all pretty much be one class, and they also are in the background.
+ GetTop is not aimed for getting versions it is for getting the first columns in a family, if you for let's say are putting the timestamp in the column name and want to fetch the most recent insert, this is when you would use GetTop instead of Get with versions.
+ RowUpdates are carrying KeyValues, but the way I have done it now, this is transparent to the user and is done at commit time, could definitely be changed.
+ newGet, see above.
+ Not really sure what you mean with that Family is being used as a byte[] on the server side. I totally agree that we should try to send as few bytes as possible over the wire, but to me it
isn't that obvious that including the family in every byte[] together with the columnName will be less than putting the familyName once and only putting the columns in a list and wrapping this in an object will be bigger.
+ Are you referring to isDeleted? Yes, this code could live somewhere else, but it is just that it has state, so then you need to have state where it lives. Just seemed easy to have all the things that are related to the compareTo(KeyValue) live in the same place for now, since they all interact with each other.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Attachments: hbase-1304-v1.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710436#action_12710436 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
I think everyone is in agreement about the future. Any stamp given on anything (put or delete) that is > now, becomes now.
As far as a Delete at LATEST, yes right now it needs to be an exact match. So, a Delete at LATEST will likely not do anything. Delete is for a single version. DeleteColumn is for all versions <= a specified stamp. So if you want to delete the latest version of something, you don't need to know it's stamp as long as you don't mind clearing all versions before it. Any time you want to delete _only_ a single version, you must know and specify the exact stamp. I think that is fair and understandable, to do otherwise I fear will sacrifice some of the good properties.
Perhaps we can figure a way to do a single-version delete of the latest version of a column without affecting older ones. But it seems you'll have to either actually perform some kind of Get during the delete to figure the stamp, which there is not currently any of outside of touching Memcache, or all of our DeleteTracker'ing will need to change how it handles Deletes.
Thoughts from others?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710448#action_12710448 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
Personally I'm a little bit confused about the whole concept of deleting the latest put, would you do this to get access the the second to last entry or when and how would this be used?
When it comes to finding the timestamp for a put maybe we can return that from the server, so that if you wanted to keep track of latest you have that option even if you don't set an explicit ts client side.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v7.patch
Full implementation of Gets, Scans, Puts, and Deletes.
Lots more unit tests on the way, there are a number of new ones already in this patch. I have also modified all existing tests to use the new API. Internally, most things are moved to the new API (all scans, puts, and deletes) but some gets remain, that's next.
Have done a lot towards removing col:fam notation throughout, still some here and there. Also removed more and more HSK references.
This patch applies cleanly to current trunk and should compile. Non-cluster unit tests are all working but there are still some bugs preventing the cluster from coming all the way up so we've not run those tests yet.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v5.patch
Applies cleanly against latest trunk and compiles. No other changes.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707574#action_12707574 ]
stack commented on HBASE-1304:
------------------------------
I skipped review of KV since I've done this earlier
ColumnCount is all ^M. Please remove them. Same for ColumnTracker, etc.
Is CC supposed to be threadsafe? If so, change counter to be AtomicInteger
Why are data members in this class public?
This import in ColumnTracker looks unused: +import org.apache.hadoop.hbase.regionserver.QueryMatcher.MatchCode;
The below should be set on the datamembers rather than in the Constructor:
{code}
+ public DeleteTracker() {
+ this.deletes = null;
+ this.delete = null;
+ this.newDeletes = new ArrayList<KeyValue>();
{code}
Why doesn't DeleteTracker#isDeleted just take a KV rather than all of its vitals?
What is a DeleteTracker? No class comment. What is a familyStamp?
newDeletes doesn't seem like a good name. New relative to what?
What is iterator a data member? I don't see where its initialized. It looks broke.
This class probably doesn't compile? I see a data member referenced as delete and as deletes.
Javadoc chart should be inside <pre></pre> else formatting will be lost.
I'll skip this class till its complete.
No javadoc on class ExplicitColumnTracker . Skiipping... Don't know what it does.
Is this a regression?
{code}
- return kv.matchingColumnNoDelimiter(this.col, this.familylength);
+ return kv.matchingColumnNoDelimiter(this.col);
{code}
Replace...
{code}
+ if(get.numFamilies() > 0) {
{code}
with get.isEmpty()? isEmpty might run faster?
Yeah, replace this:
{code}
+ for(Map.Entry<byte[],TreeSet<byte[]>> entry : get.entrySet()) {
{code}
with get.getFamily().getEntrySet()
Lets reference NavigableSet rather than TreeSet as in below:
{code}
+ for(Map.Entry<byte[],TreeSet<byte[]>> entry : get.entrySet()) {
{code
This looks wrong in HRegion:
{code}
+ this.comparator.getRawComparator());
{code}
We should be checking we are passing right comparator.
Whats this do in the Memcache.get:
{code}
+ matcher.update();
{code}
This looks odd in internalGet:
{code}
+ case SKIP:
+ break;
+ case NEXT:
+ return false;
{code}
Should NEXT be getting the next in the set? SKIP means skip out? Above they both return same answer. Is this what is wanted?
More to follow.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Attachment: HBASE-1304-ryan.patch
here are some correctness comments via a patch that layers on top of your patch indicating the areas and fixes that i think should be.
i also pushed this to the aforementioned git hub.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709895#action_12709895 ]
stack commented on HBASE-1304:
------------------------------
the praise is genuine. I love unit tests especially when not those awkward to debut integration tests that make up the bulk of our tests.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Attachment: HBASE-1304-supp.patch
late breaking ruby fix for shell
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, HBASE-1304-supp.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710324#action_12710324 ]
ryan rawson commented on HBASE-1304:
------------------------------------
I have done some recent work layering on top of v7:
http://github.com/ryanobjc/hbase/commits/hbase-1304
This fixes the scan bugs I was mentioning, look at StoreScannerTest - this test used to fail at line 76, but does not anymore.
I have more work to be done, including filters (some ideas there), and eventually bloom filters (will need help in the heap to make it happen I think) - bf is more a prototype, we wont likely get it until 0.21.
A scan won't be as fast as a get, for 2 main reasons:
- have to seek every hfile at init time
- once query matcher indicates we are 'done', we still have to continue consuming key values until the next row. We could flag this and skip this, but the 'scanner' would be useless for follow up queries.
I havent thought about the get() case nearly as much as the scan, so I don't have any ideas for filters, etc there yet. Do you?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Holstad updated HBASE-1304:
--------------------------------
Attachment: hbase-1304-v1.patch
This is a rather big patch with a lot of changes but will try to break it down as good as I can.
Two big components, the client and the server side, client first:
On the client I have added a couple of classes that will take care of querying and adding of data.
The classes for adding are:
RowUpdates, Family where the former wraps the later. The Family class is a container for the columns and values that you want to
add for that family. These are converted to KeyValues when sent to the server.
Classes for getting are:
AbstractGet, common methods for all get classes
Family, same as above but without the value, since only getting
Get, interface
GetRow, getting everything from a row
GetFamilies, getting everything from a list of families
GetColumns, getting everything from a list of families where specific columns are specified
GetTop, getting the last n inserts from every family from a list of families
TimeRange, every Get includes a TimeRange for the KeyValues you want to fetch
In every get you can also specify a filter, this has not been implemented yet, but is going to be used for the user to make a custom filter
that will be used on the server.
On the server side I have mirroring GetClasses, that basically implements one method, the compareTo(KeyValue, multiFamily) which is different for all the different calls. The notion of a multiFamily is introduced to make the comparison faster for all compares between KeyValue today for fetches, since all KeyValues share the same family, but still keep it open for the future.
New classes on server:
AbstractServerGet,
Deletes, is a class the contains all deletes + a timestamp for a deleteFamily if one is found
ServerGet
ServerGetColumns
ServerGetFamilies
ServerGetTop
There is no ServerGetRow, since it is only a GetFamilies for all and are dealt with on the server
Most of the things have been tested individually but I get problem when trying to send data between the client and the server as shown in src/test/../client/TestNewHTable.java. Think it is not a big problem but haven't been able to deal with is since I don't really understand what is needed for the rpc to work. All the classes that I are sending back and forth are "hopefully" implementing Writable and are tested in src/test/.../TestSerialization.java
Also updated KeyValue to be Writable since it is going to be the return format.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Attachments: hbase-1304-v1.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v4.patch
Fixed patch path.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709709#action_12709709 ]
stack commented on HBASE-1304:
------------------------------
Tests look great (especially bit where pieces are tested individually instead of as part of a minihbasecluster spinup). In fact, they are beautiful. I love the fact that they go on and on.
This is a little odd:
+ private final boolean PRINT = false;
Just use a logger and emit output. Who cares.
These names on DeleteCompare seem odd but maybe they'll make sense when I get to the class:
INCLUDE_NEW_NEXT_NEW
NEXT_NEW
Whats going on here?
{code}
+ List<KeyValue> l1 = new ArrayList<KeyValue>();
+ l1.add(new KeyValue(row1, fam1, col5, data));
+ l1.add(new KeyValue(row2, fam1, col1, data));
+ l1.add(new KeyValue(row2, fam1, col2, data));
+ scanners.add(new Scanner(l1));
{code}
Scanners take a list of KVs? Even though they have different rows?
Not important, just a thought --> Should KeyValueScanner implement java Queue and be called KeyValueQueue altogther? next method becomes poll. The unimplementables such as remove throw unimplemented exceptions.
Is there anything left in InternalScanner other than close?
I'm about 1/3rd of the way through. Thats enough for now. Will return to it in morning.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Resolution: Fixed
Release Note:
New API - Scan, Get. New result class 'Result'.
Some old API methods may have been removed.
Old filters are semi-broken - any row-based filtering works, but complex column filtering options don't.
Status: Resolved (was: Patch Available)
Commited, thanks to holstad,jgray,rawson,stack.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-fix.patch, HBASE-1304-ryan.patch, HBASE-1304-supp.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-1304:
------------------------------------
Assignee: Jonathan Gray (was: Erik Holstad)
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710453#action_12710453 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
My opinion is, if you care about what the stamps are, then you should set them manually. We could certainly figure out a way for a Put to return a long, but could be very tricky considering a single put could be built with a different timestamp for each column.
And what you're saying about the deleting of latest put, Erik, I agree that this is actually not a particularly common use case and it should be sufficient to allow explicit deletion by stamp. Any time you are mucking with the version-level, you must be aware of the version dimension (the timestamp). Since many use cases do not involve versions, I don't want to pollute everything else to allow for something like explicitly deleting LATEST but no others before it.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1304:
-------------------------
Fix Version/s: 0.20.0
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707839#action_12707839 ]
stack commented on HBASE-1304:
------------------------------
Thinking on it more, we should keep IBW and not make Bytes Writable. Lets not pollute util with io and MR stuff.
Whats missing from IBW is passing a byte array with offset and length and NOT making a copy in the IBW constructor; only pull on the buffer when we serialize.
I wonder if such a thing would even work in a MR context? Can we just change the IBW behavior internally?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703528#action_12703528 ]
stack commented on HBASE-1304:
------------------------------
Couple of comments:
Is multifamily a boolean that says whether a HCD has more than one? Can't client just count the families in an HCD rather than set an attribute?
>From HConstants:
"Max length a family can have because of the limitation in TFile." Shouldn't this define be in KeyValue if it isn't already?
Whats happening in KeyValue? Seems like most of added code is stuff already in KV? There are conflicts that need resolving. Looks like in most cases you want to keep what is in trunk.
This method:
{code}
+ public void set(final KeyValue kv){
+ this.bytes = kv.getBuffer();
+ this.offset = kv.getOffset();
+ this.length = kv.getLength();
+ }
{code}
is a duplicate of a copy constructor from earlier in KV.
newCommit is not the best name for a method; can we think of something else. Takes different arg. so maybe just call it commit?
GetColumns vs. GetFamilies, vs. GetRow: why have 3 types? Why ain't a get of all on a row a Get with no family spec? Why ain't a get of a column a Get with an explicit column spec? And a get on a family, a get with a family as opposed to column spec? Why not a single Get class?
What is GetTop? There is no class comment explaining it? How does this differ from get n versions?
RowUpdate should be carrying KeyValues?
newGet is not going to work. We'll be stuck with this method name for ever.
Looks like we start using byte [] for family when we get serverside. Family class just client side -- and sent over the wire -- only to be made into a byte []? Should be sent as byte [], not as Object? We should be slim as possible in RPC -- byte arrays where we can -- to save on net traffic
Seems like bunch of isDelete code belongs in comparators?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Attachments: hbase-1304-v1.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Status: Patch Available (was: Open)
this is ready to go into SVN finally. Some tests have been disabled.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v6.patch
v6 patch adds Scanners. Some unit tests included, more in the works, end to end testing coming. Anyone interested in helping out, this patch now contains full implementations of Gets and Scans, end to end. Filters have been removed from scanner logic for the time being, this obviously needs to be put back but we need revisit them.
Note, this patch will NOT compile (v5 does). I am still in the process of moving internal code to the new scanners.
Tomorrow afternoon I will post a fresh patch that applies cleanly to trunk, compiles, and adds Puts and Deletes. I may not be able to completely remove all the old gets() by then but will try.
Compactions are very easy now, they will work seamlessly with the new scanner implementations.
So for now the best way to help is to review the code in this patch, especially scanners. There are major changes to how things work, but I hope the new implementation is easier to follow.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710431#action_12710431 ]
Jim Kellerman commented on HBASE-1304:
--------------------------------------
> ryan rawson commented on HBASE-1304:
> ------------------------------------
> A few thoughts:
>
> - It never makes sense to 'delete into the future' - imagine if you
> inserted data that was masked by a delete placed there X
> days/months/years before. It would be confusing.
I agree. +1 on not deleting into the future.
The exception to this rule is LATEST_TIMESTAMP which HRegion converts
to 'now'. If I read this (somewhat lengthy) discussion correctly, any
timestamp specified for a delete that does not exactly match a
timestamp for a cell, that delete will be ignored. Converting
LATEST_TIMESTAMP to 'now' will break if my reading is correct.
> In the end, I'm concerned about minor compactions removing deleted
> key/values - are we sure we want this behaviour?
I am not concerned with removing deleted key/values on a minor
compaction because as Erik points out, there is no way to undo a
delete. The delete record still needs to be written so that key/values
from older stores can be removed (along with the delete record) during
a major compaction.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708539#action_12708539 ]
stack commented on HBASE-1304:
------------------------------
Should we add define for non-lockid of -1L?
{code}
+ private long lockId = -1L;
{code}
We going to use utility method for this?
{code}
+ public void deleteColumn(byte [] column) {
+ int len = column.length;
+ int i=0;
+ byte b = 0;
+ for(; i<len; i++) {
+ b = column[i];
+ if(b == ':') {
+ break;
+ }
+ }
..
{code}
Sorry, every time I read these Get, Delete, Result, classes, I want to put them into the client package. Pains me that they are in 'generic' io but I suppose it makes sense?
Ain't QueryMatcher the server-side version of Get? Does that mean we could move these above classes into client package?
ColumnCount is all ^Ms. ColumnTracker the same as is DeleteTracker (as do others). No biggie.
In ColumnTracer *Interface* it says this in class comment:
+ * This class is NOT thread-safe as queries are never multi-threaded
NewDeletes is not a good name. Should minimally be explained in class with a comment. Should be allocated when its defined rather than have the definition and allocation span over into Constructor.
In DeleteTracker, iterator is data member. Its created in finalize. That seems a little odd? finalize is a loaded java term. Perhaps use something else?
What is happening here?
{code}
+ public void add(byte [] buffer, int qualifierOffset, int qualifierLength,
+ long timestamp, byte type) {
+ if(type == KeyValue.Type.DeleteFamily.getCode()) {
+ if(timestamp > familyStamp) {
+ familyStamp = timestamp;
+ }
+ return;
+ }
....
{code}
Why not pass in the KV here:
{code}
+ if(timestamp > familyStamp) {
+ this.newDeletes.add(new Delete(buffer, qualifierOffset, qualifierLength,
+ type, timestamp));
+ }
{code}
... rather than make a new one to add to newDeletes? Maybe in context, passing KV is not possible?
Below looks odd. We don't seem to be adding to deletes just returning after setting familyStamp (should we be moving on the familyStamp in this way?). Whats going on here? Its not clear. Comment would help.
{code}
+ public void add(byte [] buffer, int qualifierOffset, int qualifierLength,
+ long timestamp, byte type) {
+ if(type == KeyValue.Type.DeleteFamily.getCode()) {
+ if(timestamp > familyStamp) {
+ familyStamp = timestamp;
+ }
+ return;
+ }
{code}
How does the delete datamember in DeleteTracker get set?
Rewrite this:
{code}
+ if(this.familyStamp == 0L && this.delete == null) {
+ return true;
+ }
+ return false;
{code}
Move below to top of class. Make it static?
+ protected enum DeleteCompare {
On the new Delete class, its needed really?
If internal class, I don't think you need to do below:
{code}
+ * Fields are public because they are accessed often, directly, and only
+ * within this class.
{code}
They can be private I think... at least the class should be. Make it static too.
Should DeleteCompare enums be member of Delete?
Bytes doesn't need to be Writable any more, right? Just Comparable?
Thats enough for now.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710217#action_12710217 ]
stack commented on HBASE-1304:
------------------------------
Sounds like the dictum that deletes only apply to the next file is one we could do without if we had to. But, hopefully, its one of those rules you take upon yourself to make processing cleaner (I haven't looked closely enough at code yet).
@Holstad My guess is that memcache is cleaner if deletes are applied on insertion rather than what we have now where we have to back up just in case current cell has been deleted?
I'm fine removing deleted (and expired) puts as part of minor compaction. I don't think it'll be too taxing. Obviously, the delete itself must persist in case older files have matching puts (Major compactions are the time when cells can be dropped because they exceed the maximum-versions count on the Store and its the time when you get to drop the delete cells themselves after all of the puts they overshadow have been let go).
I don't think we are doing this or even proposing doing this but just in case, +1 on not deleting into the future.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: (was: HBASE-1304-v4.patch)
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v3.patch
You now specify a KeyComparator when instantiating your QueryMatcher.
Gets will now work for catalog tables as well.
Next step is implementing Scans on top of QueryMatcher (and deps), then puts, then deletes. Compactions will, pretty much, be a special version of Scans, implemented using a heap-style merge across memcache + storefiles as discussed in other issues.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710162#action_12710162 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
@ryan, good stuff...
first, remember there are three different types of deletes. Delete (a single version of the exact specified stamp), DeleteColumn (all versions <= specified stamp), and DeleteFamily (all versions of all columns <= specified stamp).
So, if the case above is talking about a DeleteColumn of col1 @ timestamp 9, then the put @ ts=8 should be deleted, as well as any other col1s in later storefiles with ts <= 9.
When we actually do the compaction, we will process it with something like the ScanDeleteTracker, that processes the deletes as you go (merging multiple storefiles). So, the DT would actually prevent the put @ ts=8 from being output to the compacted file. However, since in this case it is DeleteColumn, we have to retain the delete in the outputted and compacted file. So we
Like I mentioned before, the difference between ScanDT and CompactDT is that the compaction one will need to make a decision about which deletes to output and which to clear. The two cases you will clear deletes from the compacted file is if it is an explicit Delete of a single version and that version exists in the merged files, or if you have a delete that overrides another delete (like a DeleteColumn of col1 @ ts =9 and @ ts = 20. The ts = 9 would not be included in the compacted file.
Alternatively, we can make Gets like Scans and then we don't need the property of deletes only applying to older ones. What we lose is the ability to early-out of a get w/o having to open and read all the storefiles. You'll never be able to just touch memcache, you'll always need to open every storefile.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Attachment: HBASE-1304.patch
this is the final patch to be committed
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710127#action_12710127 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
Dropped some thoughts on IRC, figured I'd post here:
[10:42am] jgray2: dj_ryan: i don't think v7 patch contains changes to compactions yet... not following your questions exactly but compactions need to be merged with scan code
[10:43am] jgray2: gets can be redone as scans
[10:43am] jgray2: and that's probably the direction we'll need to go
[10:43am] jgray2: if millions of columns in a single row
[10:44am] jgray2: you basically need to scan them, even within the row
[10:44am] jgray2: QueryMatcher makes the decision about what to do with a KV given the parameters of the query
[10:45am] jgray2: the two complex bits of it are a DeleteTracker and the ColumnTracker
[10:45am] jgray2: two implementations of each
[10:46am] jgray2: ScanDT and GetDT are different because, right now, a Get is not a low-level KV merge like a Scan is
[10:46am] jgray2: so when you're scanning (or compacting) you actually look at a Stores keys in strict sorted order
[10:46am] jgray2: merging all storefiles + memcache
[10:46am] jgray2: so when tracking deletes
[10:46am] jgray2: you need to track very little
[10:47am] jgray2: in a Get, you grab all keys from each storefile, starting at memcache, then going through them newest to oldest
[10:47am] jgray2: so deletes you read in one storefile will apply to any storefiles that are older
[10:47am] jgray2: so GetDT is quite a bit more complex
[10:47am] jgray2: we need to benchmark and see if scans are gooder
[10:47am] jgray2: because they are much more "correct"
[10:47am] jgray2: if you do manual timestamp setting, gets can give you indeterminate results
[10:48am] jgray2: but scans are always strictly sorted
[10:48am] jgray2: ColumnTracker is implemented as either ExplicitCT or WildcardCT
[10:48am] jgray2: explicit is when qualifiers are given, wildcard if all in a family
[10:48am] jgray2: so it tracks that, and then max versions for each
[10:49am] jgray2: honestly i've not looked at compactions since i wrote scanners but have had it in mind
[10:50am] jgray2: it will use QueryMatcher and CT/DT directly
[10:50am] jgray2: wildcardCT where maxVerisons = family setting
[10:50am] jgray2: ScanDT
[10:50am] jgray2: QueryMatcher already does TTL enforcement and such
[10:51am] jgray2: the only difference is in a minor compaction you still need to output deletes
[10:51am] jgray2: that are not fully enforced or overridden
[10:51am] jgray2: so then we'll probably have a CompactDT
[10:52am] jgray2: might need a slight modification here and there, i don't think QM is written to ever permit deletes out to the result
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707577#action_12707577 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
{quote}
Below is odd especially when Get does not implement Map
{quote}
Get.entrySet() now replaced with Get.getFamilyMap()
{quote}
Seems a pity creating a TreeMap when usual case is single family. Later we can optimize and only create the TreeMap on addition of second family?
{quote}
Yeah, there a number of additional optimizations that can be made all over the place. Will be good to see profiling, so my focus is on major algorithmic improvements first and then we can dig in further if need be. Depending how things go, we can get additional server-side optimizations by having separate client-side/server-side Get/Put/etc that share a common Writable format.
{quote}
What's this? (Result.rowResult())
{quote}
Will return existing RowResult. Commented out because I've not integrated all the code into this patch yet.
{quote}
Lots of common code. Common base abstract class?
{quote}
Which class(es) are you referring to?
{quote}
Regards TimeRange, are we for sure that most compares of timestamps are byte [] rather than long? This class is written for byte []. Surely compare of two longs is faster than creation of byte array representation and then byte compare?
{quote}
Erik liked the idea of storing byte[]'s, but that's what we could be comparing to. While writing the code, I found it so much easier to deal with longs. There is a difference between comparing byte[]'s vs converting a byte[] to a long and comparing longs directly, but again it's orders of magnitude less significant than other changes (as you suggest), so I'd rather keep it simple. I'm thinking of changing TimeRange to just storing the longs. We can change to byte[] later if shown to make a measurable difference.
Regarding Result being constructed server-side... I agree we should save wherever possible, but in this case I think a single object instantiation per query is tolerable to keep things easier to work with. Result is where our custom Writable implementation lives, which leaves the door open for further optimizations (reading all KeyValues into one big byte[] rather than single byte[] allocation per KV). We also might have other things to attach to it in the future. That being said, it's about three lines of code to change this and I don't feel that strongly either way.
+1 on all other formatting/spacing/refactor changes, will do.
Thanks a bunch for the review stack. Keep it coming.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v2.patch
Contains end to end implementation of a Get.
Most of the server-side stuff can be fully reused for Scans, etc.
Some unit testing has been done, but NOT fully tested yet. Would like to get feedback on the design before continuing further.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Erik Holstad
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Attachment: (was: HBASE-1304-supp.patch)
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, HBASE-1304-supp.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710129#action_12710129 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
CT and DT interfaces have .update() and .reset() methods. Update is called in Get operations when you move from one storefile to another (deletes apply to older storefiles only, so they are read into another array and then a sorted merge is done in update()). Reset is called in Scan operations in between rows. No enforcement of anything crosses row boundaries, outside of filters.
Filters need to be reimplemented (well, reintegrated at least). Anyone interested in them, take a look into the existing RowFilter interface and the implementations... There are a number of varying methods, some of which are only used in one or two implementations. At the least, I'd like to have three separate calls for row, qualifier name, and value. They are a bit more combined now. The allRemaining and other methods are also good for early-out scenarios. But we might be able to have enum return codes instead so we have fewer methods in the interface? They seem a bit complex... Would be nice if we could make the interface really simple, eventually I'd like to see that work dynamically at run-time so you don't have to compile filters in the server jars.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707566#action_12707566 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Stack
We wanted to keep Result as a client side Class, but it gets complicated when you are using them in MapReduce jobs and things like that, so that is why we added the creation of it
on the server side, do agree that we should try to not do this if we can avoid it.
TimeRanges are used for for gets and scans where you would compare the times with times that comes from KeyValues that are in bytes, the other place we compare time is on the ttl
check, this time is created once, so we don't have to do if for every compare.
I can't say how much this is going to effect total performance but I think we should try to make everything as fast as possible, don't really see the downside in comparing them as byte []
instead of longs.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707868#action_12707868 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Stack
I think we should do that too, we keep Bytes as it is, ImmutableBytesWritable we change so that to constructor that takes the byte[], offset and length, doesn't copy
and do the cipy when we actually write. I think that will work, but like you said we need to test it to see that it actually does the right thing.
So what we need to do is to remove the copy line in the constructor and add the two fields, offset and length, to the class, sound ok?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710214#action_12710214 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
Right, if you want to have a different value you just do another PUT, there is not delete. In that case, if you don't want versioning, you set max versions = 1 and on compaction older values would not get output to the compacted file. That is enforced in the ColumnTracker (Wildcard in compaction case).
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710181#action_12710181 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Ryan
The way I see it is that the fact that deletes only apply to earlier files is not something that is going to speed up the early out scenario for all cases, where it will help is when you have queries that don't need to touch files but only get data from memcache, since you don't need to process any deletes in memcache. The fact that deletes do, in the new implementation, only apply to older files is more like a bi product from the fact that deletes in memcache are immediately applied to the data in there.
If that is the right approach, that is a different story. The reason that I think that it makes sense comes from the fact that deletes take up a lot of resources and time when processing data, so I would like for them to be as efficient as possible. The best thing would be to apply them to the whole store as soon as they came in, but since that is not realistic we have to do something else.
So be deleting everything in memcache that is effected by the incoming delete we save time and space, by having less data to process and less flushes calls leading to fewer compactions of any kind.
The above reasoning might not make sense in all cases, but for a majority I think it does.
When it comes down to minor compactions, not sure if you are worried about them taking longer time than before where we "just" merged the results. If that is the case, most of the work for that merge is to find out which KeyValue should be the next, actually deleting the entries effected by a delete wouldn't add that much overhead.
What are your concerns when it comes to removing deleted KeyValues in a minor compaction, they are still going to be removed eventually and there is currently now way to undo your delete to get them back, so the way I see it they are just a burden for the system. What kinda of behaviour would you like to see?
Regards Erik
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1304:
-------------------------------
Attachment: HBASE-1304-supp.patch
supplemental patch to fix shell
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, HBASE-1304-supp.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch, HBASE-1304.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708776#action_12708776 ]
stack commented on HBASE-1304:
------------------------------
Sorry, broke your patch Jon -- I applied the Writable and HeapSize pieces from this patch so I could close two issues and fix HBASE-1411. Hope thats OK.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708362#action_12708362 ]
Jonathan Gray commented on HBASE-1304:
--------------------------------------
I'm about 2/3 done with scanners. The last bit is the hardest to deal with because of the multi-leveled complexity. A scan is opened, and then nexted one row at a time. Each query uses one region at a time, and for that region, has a scanner per store (family) and for each store, a scanner for memcache and one for the storefiles. Within the storefiles, there is a scanner for each hfile. This has to be all be intelligently and efficiently merged.
The primary issue I'm dealing with now is that Memcache is not immutable, so the "next" row can change between calls. For that reason, it doesn't seem to make sense to actually make them stateful. Rather, it uses tailSet() on every next call.
Should have a patch up for Scans tomorrow. I've done some stuff for Puts and Deletes but not patch-ready yet, should be tomorrow or wednesday depending on scanner testing goes.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707587#action_12707587 ]
stack commented on HBASE-1304:
------------------------------
No class comment on QueryMatcher. What is it?
All ^Ms still. Need to remove.
Does this enum need to be public? Shouldn't it be package private? Also, doc using javadoc rather than comments:
{code}
+ // ColumnMatcher.match() return codes
+ public static enum MatchCode {
+ INCLUDE, // Include in result
+ SKIP, // Do not include in result
+ NEXT, // Move to next StoreFile
+ DONE // Query done, return
+ }
{code}
Is the comment for next right? We just made use of this enum in Memcache.
Data members should be final.
Things like this.deletes should be moved out of constructor and instead make the declaration and assignment happen in the one statement.
Why not use the defines that are in KV rather than do below:
{code}
+ offset += Bytes.SIZEOF_INT + Bytes.SIZEOF_INT;
{code}
This is usually written as ret < 0 rather than as + if(ret <= -1) {
There are methods in KV for doing some of this stuff that is in match. Having it here and in KV makes it hard if we want to change KV internals. Use the KV defines or just use the KV methods getKeyLength, getRowOffset, etc. There are even versions where you can pass stuff you've already calculated -- lengths and offsets -- so the KV method doesn't do it again.
Where do we accumulate deletes? In this match I don't see that we are adding to the deletes structure.
Below could be written as return timestamp < oldestStamp instead of as:
{code}
+ if(timestamp < oldestStamp) {
+ return true;
+ }
+ return false;
{code}
If expired and memcache, do you just want to remove it? It'd be rare case I suppose?
Could we early out here:
{code}
+ // Read from Memcache
+ this.memcache.get(matcher, result);
{code}
We don't check the return from this.memcache.get?
Rather than do this.storefiles.size() == 0, do isEmpty. storefiles is a concurrentskipmaplist. size is expensive on this structures.
Do we have to make the List<HFileScanner> storefileScanners? Can't we just process the store files one at a time as they come out of the descendingMap? Whats advantage of preallocating scanners? They are used once only anyways? Just do just-in-time allocation close to where they are used?
What is a StoreFileScanner? Members should be final.
Remove the ^Ms.
Again SKIP and NEXT seem off.
Isn't it possible that INCLUDE might want us to keep going in the store file? Perhaps more answers to get (same up in memcache). Instead we break on INCLUDE.
What is a WildcardColumnTracker
Bytes is a Writable? Whats difference between Bytes and ImmutableBytesWritable? Why is it Writable? Should we deprecate ImmutableBytesWritable and use this instead?
Should use Bytes.toString() rather than new String when doing the toString for Bytes. The former does utf8. Latter whatever the user locale.
parseColumn doesn't belong in Bytes. Put it in KeyValue?
There do not seem to be any new tests?
What do we have here? Does it work? What more is to be done? What you have for an estimate before you have it all put back together? I see deletes still needs to be worked through, I see scanners and puts still need to be done. Seems like a bunch of work still to do? What you reckon lads?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710167#action_12710167 ]
ryan rawson commented on HBASE-1304:
------------------------------------
A few thoughts:
- It never makes sense to 'delete into the future' - imagine if you inserted data that was masked by a delete placed there X days/months/years before. It would be confusing.
- Given that previous, deletes are scoped to 'now' or before, given the following facts:
-- Deletes remove puts from the memcache immediately
-- No compactions
- Given these 3 facts, a delete would apply only to previous files. This fulfills Jon's previous needs for get early out.
Now, under the simple minor compaction case, where we just merge all the keys without any delete processing, we run into the scenario whereby:
- Deletes now potentially to the current file
Now, a scan must process every file because it needs to get to the next row. But a get is a specialized scan, whereby once we have fulfilled the needs of the current query we don't have to next() the rest of the store files/hfiles (e: jon's early out). For example, let's says we are looking for the columns:
- A,B - 1 version
Once we have fulfilled those requirements we can stop looking in files. Right now scanners will seek all hfiles at the same time which is something that isn't strictly necessary in a 'get()'. Early outs prevent us from scanning the rest of the files, gaining speed.
Now, I'm not sure exactly how by having the 'delete applies only to earlier files' helps the early outs - since no matter what, deletes always come before the puts they apply to even in the same file.
In the end, I'm concerned about minor compactions removing deleted key/values - are we sure we want this behaviour?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1304:
---------------------------------
Attachment: HBASE-1304-v4.patch
HBASE-1304 v4 CHANGELOG
- Added unit tests for both ColumnTrackers and DeleteTracker. Begun work on ColumnMatcher unit test.
- Fixes toString methods to ruby-style for Delete, Get, Put, Result, Scan, TimeRange
- TimeRange is now all longs, for now
- All ^M newlines should be removed, let me know if not
- Added class comments for ColumnTracker and implementors ExplicitColumnTracker/WildcardColumnTracker
- Added class comments for QueryMatcher and javadoc comments for MatchCode enum
- Cleaned up small methods to return (x < y) rather than if branches
- Added Get.hasFamilies() which wraps !Map.isEmpty()
- Made QueryMatcher.MatchCode package private
- Changed to storefiles.isEmpty instead of expensive .size()
- parseColumn moved to KV from Bytes
Result
- Added null checks
- Cleaned spacing all over the place
- Cleaned up .rowResult() and .isEmpty()
DeleteTracker
- Added class comment for DeleteTracker
- Two bugfixes for DeleteTracker found from unit tests
- DeleteTracker.add now takes individual fields rather than just a KeyValue
- DeleteTracker.Delete private class is being used to store fields, no more KV parsing in DeleteTracker
- There are a number of potential approaches to further optimize DeleteTracker. Isolating this behavior to a class should make it easier for everyone to profile/benchmark/improve it down the road.
- Currently working on extensive unit testing for this class.
- Took initial assignment out of constructor and set directly to data members.
- Iterator initialized correctly now.
@stack
- CC and others do not need to be thread-safe, yet. We have no multi-threaded Get/Scans.
- MatchCode is a return type of ColumnTracker.checkColumn, it is not unused.
- DeleteTracker now takes even more vitals vs a KeyValue. Why should we reparse KeyValues when the work has already been done in the caller?
- Regression you mention is only an inefficiency, it will be completely removed once this is finished (it's for old ColumnMatcher)
- Reason for NavigableSet vs TreeSet? Don't need navigable, just want to TreeSet stuff, and that's what it is. Result makes extensive use of TreeSet, changing to NavigableSet is going to make things even wilder looking. No strong opinion, just wondering what you think (I like to keep things short) :)
- Why this.comparator.getRawComparator() looks wrong in HRegion? I'm passing the regions raw comparator, we can add a check but seems there are serious things wrong if a Region is instantiated with an incorrect comparator. Didn't we want to try to remove checks all over the place for isMeta type things?
- Added comments to QueryMatcher, should answer your questions.
- After we do some more testing, I will switch KeyValue parsing to utilize existing methods in KV class rather than doing inline. Will also use < 0 rather than <= -1 as you suggest.
- Deletes are accumulated with DeleteTracker.add
- You are right about earlying out from Memcache.get(), I was returning to do that but not handling properly in Store.get
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710183#action_12710183 ]
ryan rawson commented on HBASE-1304:
------------------------------------
I'm just thinking about what the usage scenario for deletes are - ultimately if you want a different value instead of the original, you are better off to just insert over, rather than to delete then put again.
Deleting immediately from the memcache is fine with me.
So the question is, do we want the complexity of deletes in the scan/get, or do we want to take the higher hit on compactions - potentially. I don't think compaction & deletion at the same time is overly complex, and if it makes the scan/get simpler, then perhaps it would be best.
So far I don't think the scan delete tracker works - can you confirm/deny?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710153#action_12710153 ]
stack commented on HBASE-1304:
------------------------------
FYI, Ryan put up trunk+1304v7 here: http://github.com/ryanobjc/hbase/tree/hbase-1304 Lets work off here going forward?
Also, from #hbase:
20:47 < dj_ryan> jgray
20:47 < dj_ryan> so my thought is
20:47 < dj_ryan> if you say that deletes always apply to later files
20:47 < dj_ryan> for optimization in gets
20:47 < dj_ryan> when you minor compact you may run into the situation where deletes apply to puts from the same file
21:16 < dj_ryan> here's what i am thinking
21:17 < dj_ryan> lets say you have (for the same, row, cf, cq)
21:17 < dj_ryan> delete, 6
21:17 < dj_ryan> then
21:17 < dj_ryan> no
21:17 < dj_ryan> start over
21:17 < dj_ryan> row1,fam,col1, 10, put, X
21:17 < dj_ryan> row1,fam,col1, 9, delete
21:17 < dj_ryan> next file:
21:18 < dj_ryan> row1,fam,col1,8,put
21:18 < dj_ryan> <end>
21:18 < dj_ryan> ok so this is great
21:18 < dj_ryan> the delete applies to the 2nd file
21:18 < dj_ryan> but during a minor compaction we end up with:
21:18 < dj_ryan> row1,fam,fol1, 10, put x
21:18 < dj_ryan> row1,fam,col1, 9, delete
21:18 < dj_ryan> row1, fam, col1, 8, put
21:18 < dj_ryan> and now we have a situation where the delete applies to a put of the same file
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707833#action_12707833 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Stack.
Yeah, sounds good to deprecate Immutable and keep Bytes in util. I'm just a little bit confused why we should be careful to use it. For the MR jobs we still need something that wraps a byte[], no?
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1304) New client server implementation of
how gets and puts are handled.
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710398#action_12710398 ]
Erik Holstad commented on HBASE-1304:
-------------------------------------
@Ryan
Nice job!
I agree that we can't early out when we hit deleteFamily since we don't know if there will be any DeleteColumn and Delete that is newer, is that what you are saying too?
DeleteFamily doesn't have to be first on the row though, there might be some columns with the null qualifier that sits before.
Don't really follow why the seek() method will not work.
The way I saw the problem with HBASE-867 was that we had two problems:
1. Scanning was too slow in general, since if you asked for all columns and not explicit ones that query would time out too, which is really bad.
2. When asking for explicit column you would still have to next through all of them even if you had them all for the current row, this is what seek() is trying to help out with, by going back to the file and ask for the next row, so we don't have too next through many blocks.
The thing with filters and gets is more of a Filter design issue. But I think that if we rewrite Filters to fit them new implementation better I see no problem in putting them in there, after all this is one of the things that we wanted to accomplish with this rewrite.
> New client server implementation of how gets and puts are handled.
> -------------------------------------------------------------------
>
> Key: HBASE-1304
> URL: https://issues.apache.org/jira/browse/HBASE-1304
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Erik Holstad
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1304-ryan.patch, hbase-1304-v1.patch, HBASE-1304-v2.patch, HBASE-1304-v3.patch, HBASE-1304-v4.patch, HBASE-1304-v5.patch, HBASE-1304-v6.patch, HBASE-1304-v7.patch
>
>
> Creating an issue where the implementation of the new client and server will go. Leaving HBASE-1249 as a discussion forum and will put code and patches here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.