You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by renato2099 <gi...@git.apache.org> on 2016/08/31 11:35:10 UTC
[GitHub] gora pull request #86: GORA-443 Upgrade HBase to 1.2.0
GitHub user renato2099 opened a pull request:
https://github.com/apache/gora/pull/86
GORA-443 Upgrade HBase to 1.2.0
Work in progress for updating gora to 1.2.2. I also looked at @lewismc 's work ;)
All of tests pass when run individually, but when run as a whole some of them fail. I think this is because of the BufferedMutator and taking longer than it should. The usage of the BufferedMutator is like setting autoFlush=False which then makes sense to actually use our Flush method.
The weird thing is that if we use the regular Table and do the operations synchronously all tests pass always. This would be like setting autoFlush=True. The problem with using this option is that Gora wouldn't be batching any operations and just passing them down.
Maybe someone has an idea on how to check for BufferedMutator to be done (?)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/renato2099/gora GORA-443
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/gora/pull/86.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #86
----
commit 01856b565be8b6e890130dedc3bed89f63c96daa
Author: Renato Marroquin <ma...@inf.ethz.ch>
Date: 2016-08-31T11:26:51Z
GORA-443
commit b125b800187be7304461b6ed25a4288c3c58cd05
Author: Renato Marroquin <ma...@inf.ethz.ch>
Date: 2016-08-31T11:28:39Z
Merge branch 'master' into GORA-443
commit 163d91bc284e5b5a53b9d314e9fd317964cef486
Author: Renato Marroquin <ma...@inf.ethz.ch>
Date: 2016-08-31T11:33:13Z
GORA-443
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #86: GORA-443 Upgrade HBase to 1.2.0
Posted by renato2099 <gi...@git.apache.org>.
Github user renato2099 commented on a diff in the pull request:
https://github.com/apache/gora/pull/86#discussion_r77372183
--- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java ---
@@ -57,13 +57,7 @@
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.client.ResultScanner;
-import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.*;
--- End diff --
For sure, I will do it for the final patch. There is still some clean up to be done ;)
what I'd like to fix is this weird behaviour of BufferedMutator and our tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #86: GORA-443 Upgrade HBase to 1.2.0
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/gora/pull/86
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #86: GORA-443 Upgrade HBase to 1.2.0
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/gora/pull/86#discussion_r78299720
--- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseTableConnection.java ---
@@ -83,288 +87,129 @@
public HBaseTableConnection(Configuration conf, String tableName, boolean autoflush)
throws IOException {
this.conf = conf;
+
this.tables = new ThreadLocal<>();
+ this.buffers = new ThreadLocal<>();
+ this.connection = ConnectionFactory.createConnection(conf);
this.tableName = TableName.valueOf(tableName);
+ this.regionLocator = this.connection.getRegionLocator(this.tableName);
+
this.autoFlush = autoflush;
}
-
- private HTable getTable() throws IOException {
- HTable table = tables.get();
+
+ private Table getTable() throws IOException {
+ Table table = tables.get();
if (table == null) {
- table = new HTable(conf, tableName) {
- @Override
- public synchronized void flushCommits() throws RetriesExhaustedWithDetailsException, InterruptedIOException {
- super.flushCommits();
- }
- };
- table.setAutoFlushTo(autoFlush);
- pool.add(table); //keep track
+ table = connection.getTable(tableName);
+// table.setAutoFlushTo(autoFlush);
+ tPool.add(table); //keep track
tables.set(table);
}
return table;
}
-
- @Override
+
+ private ConcurrentLinkedQueue<Mutation> getBuffer() throws IOException {
+ ConcurrentLinkedQueue<Mutation> buffer = buffers.get();
+ if (buffer == null) {
+// BufferedMutatorParams params = new BufferedMutatorParams(this.tableName).listener(listener);
+// buffer = connection.getBufferedMutator(this.tableName);
+ buffer = new ConcurrentLinkedQueue<>();
+ bPool.add(buffer);
+ buffers.set(buffer);
+ }
+ return buffer;
+ }
+
+ public void flushCommits() throws IOException {
+ BufferedMutator bufMutator = connection.getBufferedMutator(this.tableName);
+ for (ConcurrentLinkedQueue<Mutation> buffer : bPool) {
+ for (Mutation m: buffer) {
+ bufMutator.mutate(m);
+ bufMutator.flush();
--- End diff --
This can be lifted outside the for loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #86: GORA-443 Upgrade HBase to 1.2.0
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/86#discussion_r78889929
--- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseTableConnection.java ---
@@ -83,288 +87,129 @@
public HBaseTableConnection(Configuration conf, String tableName, boolean autoflush)
throws IOException {
this.conf = conf;
+
this.tables = new ThreadLocal<>();
+ this.buffers = new ThreadLocal<>();
+ this.connection = ConnectionFactory.createConnection(conf);
this.tableName = TableName.valueOf(tableName);
+ this.regionLocator = this.connection.getRegionLocator(this.tableName);
+
this.autoFlush = autoflush;
}
-
- private HTable getTable() throws IOException {
- HTable table = tables.get();
+
+ private Table getTable() throws IOException {
+ Table table = tables.get();
if (table == null) {
- table = new HTable(conf, tableName) {
- @Override
- public synchronized void flushCommits() throws RetriesExhaustedWithDetailsException, InterruptedIOException {
- super.flushCommits();
- }
- };
- table.setAutoFlushTo(autoFlush);
- pool.add(table); //keep track
+ table = connection.getTable(tableName);
+// table.setAutoFlushTo(autoFlush);
+ tPool.add(table); //keep track
tables.set(table);
}
return table;
}
-
- @Override
+
+ private ConcurrentLinkedQueue<Mutation> getBuffer() throws IOException {
+ ConcurrentLinkedQueue<Mutation> buffer = buffers.get();
+ if (buffer == null) {
+// BufferedMutatorParams params = new BufferedMutatorParams(this.tableName).listener(listener);
+// buffer = connection.getBufferedMutator(this.tableName);
+ buffer = new ConcurrentLinkedQueue<>();
+ bPool.add(buffer);
+ buffers.set(buffer);
+ }
+ return buffer;
+ }
+
+ public void flushCommits() throws IOException {
+ BufferedMutator bufMutator = connection.getBufferedMutator(this.tableName);
+ for (ConcurrentLinkedQueue<Mutation> buffer : bPool) {
+ for (Mutation m: buffer) {
+ bufMutator.mutate(m);
+ bufMutator.flush();
--- End diff --
OK thanks. I've pushed a PR to @renato2099 which I hope he will add to this one. You can see the PR at https://github.com/renato2099/gora/pull/1
@renato2099 can you please merge and push the combined PR to this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #86: GORA-443 Upgrade HBase to 1.2.0
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/86#discussion_r77371850
--- Diff: gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java ---
@@ -57,13 +57,7 @@
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.HTableDescriptor;
-import org.apache.hadoop.hbase.client.Delete;
-import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HBaseAdmin;
-import org.apache.hadoop.hbase.client.Put;
-import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.client.ResultScanner;
-import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.*;
--- End diff --
Can you import individual classes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #86: GORA-443 Upgrade HBase to 1.2.0
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/86
@renato2099 can you please merge and push https://github.com/renato2099/gora/pull/1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #86: GORA-443 Upgrade HBase to 1.2.0
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/86
I notice that we never ```.close()``` Admin. According to @enis we should http://www.slideshare.net/enissoz/meet-hbase-10/38?src=clipshare
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #86: GORA-443 Upgrade HBase to 1.2.0
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on the issue:
https://github.com/apache/gora/pull/86
After using admin, .close() method should be called.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---