You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by bhupeshchawda <gi...@git.apache.org> on 2015/12/31 09:49:31 UTC
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
GitHub user bhupeshchawda opened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/155
[Review only] MLHR 1957 hbase input
@sandeepdeshmukh
Can you please review?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bhupeshchawda/incubator-apex-malhar MLHR-1957-hbase-input
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-malhar/pull/155.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #155
----
commit a1fe2af77ee8e5572e28a5eccb26a6394f30878d
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-24T13:20:25Z
Added threading for reading data from hbase
commit 25c580f24fb5cfece5dadee15bbb0b8aab4bea2f
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-28T12:18:53Z
Intermediate commit. WIP
commit 5f05526d1fa117eadaf31f111ab8dcf60191ca75
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-29T07:11:56Z
Fixed problems. Test case running OK
commit de50218c8e74d593cf2ee5d95426f78df1b529b6
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-29T08:41:09Z
Added metrics
commit 86e9a27a28c09dec1d1f37f0e76cf08339c81cfa
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-30T13:35:28Z
Fixed class hierarchy. Fixed unit test
commit 35bcae19e0b5571f5ec1b4c46374f60a431def30
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-30T14:11:03Z
Fixed variable scoping. Adhoc changes
commit 11d64accdda166dd6f4c78545b997e6a655f928d
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-31T06:42:38Z
Adhoc changes
commit b17f61f22c19b32e0b7112a068f6150a04790d90
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-31T08:43:28Z
Adhoc changes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r49044564
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseFieldValueGenerator.java ---
@@ -0,0 +1,46 @@
+package com.datatorrent.contrib.hbase;
+
+import java.util.List;
+
+import com.datatorrent.lib.util.FieldValueGenerator;
+import com.datatorrent.lib.util.PojoUtils;
+
+/**
+ * A {@link FieldValueGenerator} implementation for {@link HBaseFieldInfo}
+ */
+public class HBaseFieldValueGenerator extends FieldValueGenerator<HBaseFieldInfo>
+{
+
+ public static HBaseFieldValueGenerator getHBaseFieldValueGenerator(final Class<?> clazz, List<HBaseFieldInfo>
--- End diff --
Actually that is a static method, so could not override it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by tushargosavi <gi...@git.apache.org>.
Github user tushargosavi commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48653237
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseFieldValueGenerator.java ---
@@ -0,0 +1,46 @@
+package com.datatorrent.contrib.hbase;
--- End diff --
license headers missing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: MLHR 1957 hbase input
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/155
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
GitHub user bhupeshchawda reopened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/155
[Review only] MLHR 1957 hbase input
@sandeepdeshmukh
Can you please review?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bhupeshchawda/incubator-apex-malhar MLHR-1957-hbase-input
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-malhar/pull/155.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #155
----
commit 236b11392afae366cb5df734945ec45fbf4b6ed7
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-24T13:20:25Z
Added threading for reading data from hbase
commit 7b94bf3a8bf4763f6b6c1d17e62cd1811a0d4bd3
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-28T12:18:53Z
Intermediate commit. WIP
commit 4e7499d4ec5ec35994cae8e76d1ee24e4cabba6a
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-29T07:11:56Z
Fixed problems. Test case running OK
commit b3291bc95a07b789d4aecf57ce2520fae8887516
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-29T08:41:09Z
Added metrics
commit e40e5f8a794784cc762b6d5662aa8d1fffabc8dd
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-30T13:35:28Z
Fixed class hierarchy. Fixed unit test
commit 30d7db03a5471e840879ad56ff31bfb3118affe6
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-30T14:11:03Z
Fixed variable scoping. Adhoc changes
commit 02eb9f71870b6122f80091c983e7e5c2a7e14a74
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-31T06:42:38Z
Adhoc changes
commit d0574439812f8795b3858cbb9cbc43dfc6bd28a3
Author: bhupesh <bh...@gmail.com>
Date: 2015-12-31T08:43:28Z
Adhoc changes
commit cbfe3b74467f395d9b23afe1f1c0d7df5356c35d
Author: bhupesh <bh...@gmail.com>
Date: 2016-01-07T06:40:11Z
Incorporating review comments
commit 278828eb7eee66f77ae9557a876fb384f2dd97c4
Author: bhupesh <bh...@gmail.com>
Date: 2016-01-22T08:30:19Z
Updated to poll continously.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by DT-Priyanka <gi...@git.apache.org>.
Github user DT-Priyanka commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48722423
--- Diff: contrib/src/test/java/com/datatorrent/contrib/hbase/HBasePOJOInputOperatorTest.java ---
@@ -59,17 +60,31 @@ public MyGenerator()
this.setTupleType( TestPOJO.class );
}
}
-
+
+ public static class TestHBasePOJOInputOperator extends HBasePOJOInputOperator
+ {
+ @Override
+ public void setup(OperatorContext context)
+ {
+ try {
--- End diff --
why do we need sleep here? A comment would be helpful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by DT-Priyanka <gi...@git.apache.org>.
Github user DT-Priyanka commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48720473
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseFieldValueGenerator.java ---
@@ -0,0 +1,46 @@
+package com.datatorrent.contrib.hbase;
+
+import java.util.List;
+
+import com.datatorrent.lib.util.FieldValueGenerator;
+import com.datatorrent.lib.util.PojoUtils;
+
+/**
+ * A {@link FieldValueGenerator} implementation for {@link HBaseFieldInfo}
+ */
+public class HBaseFieldValueGenerator extends FieldValueGenerator<HBaseFieldInfo>
+{
+
+ public static HBaseFieldValueGenerator getHBaseFieldValueGenerator(final Class<?> clazz, List<HBaseFieldInfo>
+ fieldInfos)
+ {
+ return new HBaseFieldValueGenerator(clazz, fieldInfos);
+ }
+
+ @SuppressWarnings("unchecked")
+ protected HBaseFieldValueGenerator(final Class<?> clazz, List<HBaseFieldInfo> fieldInfos)
+ {
+ for (HBaseFieldInfo fieldInfo : fieldInfos) {
+ fieldInfoMap.put(fieldInfo.getFamilyName() + ":" + fieldInfo.getColumnName(), fieldInfo);
--- End diff --
declare ":" as constant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r49044970
--- Diff: contrib/src/test/java/com/datatorrent/contrib/hbase/HBasePOJOInputOperatorTest.java ---
@@ -59,17 +60,31 @@ public MyGenerator()
this.setTupleType( TestPOJO.class );
}
}
-
+
+ public static class TestHBasePOJOInputOperator extends HBasePOJOInputOperator
+ {
+ @Override
+ public void setup(OperatorContext context)
+ {
+ try {
--- End diff --
This was added to let the output operator first insert data into hbase table in order for the input operator to read it. Will add comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48950521
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseFieldValueGenerator.java ---
@@ -0,0 +1,46 @@
+package com.datatorrent.contrib.hbase;
+
+import java.util.List;
+
+import com.datatorrent.lib.util.FieldValueGenerator;
+import com.datatorrent.lib.util.PojoUtils;
+
+/**
+ * A {@link FieldValueGenerator} implementation for {@link HBaseFieldInfo}
--- End diff --
Will add detailed comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by DT-Priyanka <gi...@git.apache.org>.
Github user DT-Priyanka commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48721128
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseInputOperator.java ---
@@ -33,8 +34,9 @@
* @param <T> The tuple type
* @since 0.3.2
*/
-public abstract class HBaseInputOperator<T> extends HBaseOperatorBase implements InputOperator
+public abstract class HBaseInputOperator<T> extends BaseOperator implements InputOperator
--- End diff --
IMO it will be better if we can implement AbstractStoreInputOperator instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by DT-Priyanka <gi...@git.apache.org>.
Github user DT-Priyanka commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48720413
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseFieldValueGenerator.java ---
@@ -0,0 +1,46 @@
+package com.datatorrent.contrib.hbase;
+
+import java.util.List;
+
+import com.datatorrent.lib.util.FieldValueGenerator;
+import com.datatorrent.lib.util.PojoUtils;
+
+/**
+ * A {@link FieldValueGenerator} implementation for {@link HBaseFieldInfo}
+ */
+public class HBaseFieldValueGenerator extends FieldValueGenerator<HBaseFieldInfo>
+{
+
+ public static HBaseFieldValueGenerator getHBaseFieldValueGenerator(final Class<?> clazz, List<HBaseFieldInfo>
--- End diff --
why don't we override "getFieldValueGenerator()"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by DT-Priyanka <gi...@git.apache.org>.
Github user DT-Priyanka commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48720278
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseFieldValueGenerator.java ---
@@ -0,0 +1,46 @@
+package com.datatorrent.contrib.hbase;
+
+import java.util.List;
+
+import com.datatorrent.lib.util.FieldValueGenerator;
+import com.datatorrent.lib.util.PojoUtils;
+
+/**
+ * A {@link FieldValueGenerator} implementation for {@link HBaseFieldInfo}
--- End diff --
Can we have better comment, this is confusing, "implementation for HBaseFieldInfo"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by tushargosavi <gi...@git.apache.org>.
Github user tushargosavi commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48653269
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseGetOperator.java ---
@@ -47,7 +47,7 @@ public void emitTuples()
{
try {
Get get = operationGet();
- Result result = table.get(get);
+ Result result = getStore().getTable().get(get);
KeyValue[] kvs = result.raw();
//T t = getTuple(kvs);
--- End diff --
Removed commented code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/155
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by sandeepdeshmukh <gi...@git.apache.org>.
Github user sandeepdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48650867
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBasePOJOInputOperator.java ---
@@ -81,104 +81,89 @@ public void setup(OperatorContext context)
}
@Override
- public void beginWindow(long windowId)
- {
- }
-
- @Override
- public void teardown()
- {
- try {
- store.disconnect();
- } catch (IOException ex) {
- throw new RuntimeException(ex);
- }
- }
-
- @Override
- public void emitTuples()
+ protected Object getTuple(Result result)
{
try {
- Scan scan = nextScan();
- if (scan == null)
- return;
-
- ResultScanner resultScanner = store.getTable().getScanner(scan);
-
- while (true) {
- Result result = resultScanner.next();
- if (result == null)
- break;
-
- String readRow = Bytes.toString(result.getRow());
- if( readRow.equals( lastReadRow ))
- continue;
-
- Object instance = pojoType.newInstance();
- rowSetter.set(instance, readRow);
-
- List<Cell> cells = result.listCells();
+ String readRow = Bytes.toString(result.getRow());
+ if( readRow.equals( getLastReadRow() )) {
+ return null;
+ }
- for (Cell cell : cells) {
- String columnName = Bytes.toString(CellUtil.cloneQualifier(cell));
- byte[] value = CellUtil.cloneValue(cell);
- fieldValueGenerator.setColumnValue( instance, columnName, value, valueConverter );
- }
+ Object instance = pojoType.newInstance();
+ rowSetter.set(instance, readRow);
- outputPort.emit(instance);
- lastReadRow = readRow;
+ List<Cell> cells = result.listCells();
+ for (Cell cell : cells) {
+ String columnName = Bytes.toString(CellUtil.cloneQualifier(cell));
+ String columnFamily = Bytes.toString(CellUtil.cloneFamily(cell));
+ byte[] value = CellUtil.cloneValue(cell);
+ ((HBaseFieldValueGenerator)fieldValueGenerator).setColumnValue(instance, columnName, columnFamily, value,
+ valueConverter);
}
+ setLastReadRow(readRow);
+ return instance;
} catch (Exception e) {
- throw new RuntimeException(e.getMessage());
+ throw new RuntimeException(e);
}
-
- }
-
- protected Scan nextScan()
- {
- if(lastReadRow==null && startRow==null )
- return new Scan();
- else
- return new Scan( Bytes.toBytes( lastReadRow == null ? startRow : lastReadRow ) );
}
- public HBaseStore getStore()
- {
- return store;
- }
- public void setStore(HBaseStore store)
+ @Override
+ protected Scan operationScan()
{
- this.store = store;
+ Scan scan;
+ if(getLastReadRow()==null && getStartRow()==null ) {
--- End diff --
Write comment on what this condition is handling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by sandeepdeshmukh <gi...@git.apache.org>.
Github user sandeepdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48650919
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBasePOJOInputOperator.java ---
@@ -81,104 +81,89 @@ public void setup(OperatorContext context)
}
@Override
- public void beginWindow(long windowId)
- {
- }
-
- @Override
- public void teardown()
- {
- try {
- store.disconnect();
- } catch (IOException ex) {
- throw new RuntimeException(ex);
- }
- }
-
- @Override
- public void emitTuples()
+ protected Object getTuple(Result result)
{
try {
- Scan scan = nextScan();
- if (scan == null)
- return;
-
- ResultScanner resultScanner = store.getTable().getScanner(scan);
-
- while (true) {
- Result result = resultScanner.next();
- if (result == null)
- break;
-
- String readRow = Bytes.toString(result.getRow());
- if( readRow.equals( lastReadRow ))
- continue;
-
- Object instance = pojoType.newInstance();
- rowSetter.set(instance, readRow);
-
- List<Cell> cells = result.listCells();
+ String readRow = Bytes.toString(result.getRow());
+ if( readRow.equals( getLastReadRow() )) {
+ return null;
+ }
- for (Cell cell : cells) {
- String columnName = Bytes.toString(CellUtil.cloneQualifier(cell));
- byte[] value = CellUtil.cloneValue(cell);
- fieldValueGenerator.setColumnValue( instance, columnName, value, valueConverter );
- }
+ Object instance = pojoType.newInstance();
+ rowSetter.set(instance, readRow);
- outputPort.emit(instance);
- lastReadRow = readRow;
+ List<Cell> cells = result.listCells();
+ for (Cell cell : cells) {
+ String columnName = Bytes.toString(CellUtil.cloneQualifier(cell));
+ String columnFamily = Bytes.toString(CellUtil.cloneFamily(cell));
+ byte[] value = CellUtil.cloneValue(cell);
+ ((HBaseFieldValueGenerator)fieldValueGenerator).setColumnValue(instance, columnName, columnFamily, value,
+ valueConverter);
}
+ setLastReadRow(readRow);
+ return instance;
} catch (Exception e) {
- throw new RuntimeException(e.getMessage());
+ throw new RuntimeException(e);
}
-
- }
-
- protected Scan nextScan()
- {
- if(lastReadRow==null && startRow==null )
- return new Scan();
- else
- return new Scan( Bytes.toBytes( lastReadRow == null ? startRow : lastReadRow ) );
}
- public HBaseStore getStore()
- {
- return store;
- }
- public void setStore(HBaseStore store)
+ @Override
+ protected Scan operationScan()
{
- this.store = store;
+ Scan scan;
+ if(getLastReadRow()==null && getStartRow()==null ) {
+ scan = new Scan();
+ }
+ else if(getEndRow() == null) {
+ scan = new Scan(Bytes.toBytes(getLastReadRow() == null ? getStartRow() : getLastReadRow()));
--- End diff --
Get this into a variable and use it later: Bytes.toBytes(getLastReadRow() == null ? getStartRow()
Readability will increase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by sandeepdeshmukh <gi...@git.apache.org>.
Github user sandeepdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48651126
--- Diff: contrib/src/test/java/com/datatorrent/contrib/hbase/HBasePOJOInputOperatorTest.java ---
@@ -128,7 +143,7 @@ public void populateDAG(DAG dag, Configuration conf)
Thread.sleep(1000);
}
catch( Exception e ){}
-
+ logger.info("Tuple row key: ", output.getReceivedTuples());
--- End diff --
make it debug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by DT-Priyanka <gi...@git.apache.org>.
Github user DT-Priyanka commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48721930
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseScanOperator.java ---
@@ -40,27 +50,97 @@
* @tags hbase, scan, input operator
* @since 0.3.2
*/
-public abstract class HBaseScanOperator<T> extends HBaseInputOperator<T>
+public abstract class HBaseScanOperator<T> extends HBaseInputOperator<T> implements Operator.ActivationListener<Context>
{
+ public static final int DEF_HINT_SCAN_LOOKAHEAD = 2;
+ public static final int DEF_QUEUE_SIZE = 1000;
+ public static final int DEF_SLEEP_MILLIS = 10;
+
+ private String startRow;
+ private String endRow;
+ private String lastReadRow;
+ private int hintScanLookahead = DEF_HINT_SCAN_LOOKAHEAD;
+ private int queueSize = DEF_QUEUE_SIZE;
+ private int sleepMillis = DEF_SLEEP_MILLIS;
+ private Queue<Result> resultQueue;
+
+ @AutoMetric
+ protected long tuplesRead;
+
+ // Transients
+ protected transient Scan scan;
+ protected transient ResultScanner scanner;
+ protected transient Thread readThread;
+
+ @Override
+ public void setup(OperatorContext context)
+ {
+ super.setup(context);
+ resultQueue = Queues.newLinkedBlockingQueue(queueSize);
+ }
+
+ @Override
+ public void activate(Context context)
+ {
+ try {
+ scan = operationScan();
+ scanner = getStore().getTable().getScanner(scan);
+ readThread = new Thread(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ Result result;
+ while ((result = scanner.next()) != null) {
+ while (!resultQueue.offer(result)) {
+ Thread.sleep(DEF_SLEEP_MILLIS);
+ }
+ }
+ } catch (Exception e) {
+ logger.debug("Exception in fetching results");
+ throw new RuntimeException(e.getMessage());
--- End diff --
better to throw entire stacktrace. Also log the exception message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r49044846
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseInputOperator.java ---
@@ -33,8 +34,9 @@
* @param <T> The tuple type
* @since 0.3.2
*/
-public abstract class HBaseInputOperator<T> extends HBaseOperatorBase implements InputOperator
+public abstract class HBaseInputOperator<T> extends BaseOperator implements InputOperator
--- End diff --
Yes. Did not know about this. Will extend it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-apex-malhar pull request: [Review only] MLHR 1957 hbase ...
Posted by bhupeshchawda <gi...@git.apache.org>.
Github user bhupeshchawda commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/155#discussion_r48950491
--- Diff: contrib/src/main/java/com/datatorrent/contrib/hbase/HBaseGetOperator.java ---
@@ -47,7 +47,7 @@ public void emitTuples()
{
try {
Get get = operationGet();
- Result result = table.get(get);
+ Result result = getStore().getTable().get(get);
KeyValue[] kvs = result.raw();
//T t = getTuple(kvs);
--- End diff --
Did not remove since not related to this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---