You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by ryaneleary <gi...@git.apache.org> on 2014/05/18 03:12:56 UTC
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
GitHub user ryaneleary opened a pull request:
https://github.com/apache/accumulo/pull/7
ACCUMULO-2825 Add RowEncodingIterator
The WholeRowIterator now extends the abstract RowEncodingIterator.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ryaneleary/accumulo ACCUMULO-2825
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo/pull/7.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7
----
commit 73a9e6cfcad6c836ed9eb95421bc7306037739a9
Author: Ryan Leary <rl...@bbn.com>
Date: 2014-05-18T01:03:03Z
ACCUMULO-2825 Add RowEncodingIterator
The WholeRowIterator now extends the abstract RowEncodingIterator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/7#discussion_r12812443
--- Diff: core/src/main/java/org/apache/accumulo/core/iterators/user/RowEncodingIterator.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.core.iterators.user;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+
+import org.apache.accumulo.core.data.ByteSequence;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.PartialKey;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.IteratorEnvironment;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.hadoop.io.Text;
+
+/**
+ *
+ * The WholeRowIterator is designed to provide row-isolation so that queries see mutations as atomic. It does so by encapsulating an entire row of key/value
--- End diff --
Would be good to update this javadoc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by ryaneleary <gi...@git.apache.org>.
Github user ryaneleary commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/7#discussion_r12844676
--- Diff: core/src/main/java/org/apache/accumulo/core/iterators/user/RowEncodingIterator.java ---
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.core.iterators.user;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+
+import org.apache.accumulo.core.data.ByteSequence;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.PartialKey;
+import org.apache.accumulo.core.data.Range;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.iterators.IteratorEnvironment;
+import org.apache.accumulo.core.iterators.SortedKeyValueIterator;
+import org.apache.hadoop.io.Text;
+
+/**
+ *
+ * The WholeRowIterator is designed to provide row-isolation so that queries see mutations as atomic. It does so by encapsulating an entire row of key/value
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the pull request:
https://github.com/apache/accumulo/pull/7#issuecomment-43654266
Cool, I think that helps a lot. That's a neat approach to reduce the amount of boilerplate between your "database" and application server. Thanks for taking the time to write this up.
I'll let this open for a little while longer if anyone else has feedback they want to give. If none comes out, I'll merge this in later tonight.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the pull request:
https://github.com/apache/accumulo/pull/7#issuecomment-43547788
@ryaneleary I know you had mentioned that you wanted this abstraction point for a reason, but I don't recall what it was. Can you elaborate on the types of operations that this abstract class provides, some use case that the WholeRowIterator didn't sufficiently provide, and, if you have it, maybe an example class?
That might help cement this for when/why a user would want to extend RowEncodingIterator
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/accumulo/pull/7
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by ryaneleary <gi...@git.apache.org>.
Github user ryaneleary commented on the pull request:
https://github.com/apache/accumulo/pull/7#issuecomment-43555087
Sure. Here's an example of what I've done (by modifying the WholeRowIterator). Assume a table that contains 'documents' and various document information. Assume a table format like this (format: ```<row <cf : cq>> = value```):
```
<docid1 <META : timestamp>> (long) 12345
<docid1 <META : meta_field_1>> (String) metaFieldValue1
<docid1 <META : meta_field_2>> (String) metaFieldValue2
<docid2 <META : timestamp>> (long) 23456
```
For other reasons in the system, I have a protocol buffer definition that can be used for shipping this data around. It's definition looks like this:
```
message DocumentMessage {
optional int64 timestamp = 1;
optional string meta_field_1 = 2;
optional string meta_field_2 = 3;
}
```
I can define an iterator that is passed a generated protocol buffer class that can iterate over the whole row and automatically build the protocol buffer object. getTopValue() now returns a protocol buffer message that the application knows how to parse.
This is nice for bulk loading and saves the client from having to build the message it would have had to build anyway. When iterating over the results returned by the scanner, an entire document at a time is returned instead of individual columns.
I can provide a more concrete example/implementation of such a class if you'd like.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request: ACCUMULO-2825 Add RowEncodingIterator
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the pull request:
https://github.com/apache/accumulo/pull/7#issuecomment-43707669
Thanks for the code, @ryaneleary!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---