You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by neoremind <gi...@git.apache.org> on 2018/03/05 14:29:52 UTC
[GitHub] flink pull request #5639: [FLINK-8862] [HBase] Support HBase snapshot read
GitHub user neoremind opened a pull request:
https://github.com/apache/flink/pull/5639
[FLINK-8862] [HBase] Support HBase snapshot read
## What is the purpose of the change
*Flink-hbase connector only supports reading/scanning HBase over region server scanner, there is also [snapshot](http://hbase.apache.org/book.html#ops.snapshots) scanning solution, just like Hadoop provides 2 ways to scan HBase, one is [TableInputFormat](https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableInputFormat.html), the other is [TableSnapshotInputFormat](https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.html), so it would be great if flink supports both solutions to ensure more wider usage scope and provide alternatives for users.*
## Brief change log
- *Create `TableInputSplitStrategy` interface and its implementations as abstraction logic for `AbstractTableInputFormat`*
- *Update `HBaseRowInputFormat` and `TableInputFormat`*
- *Add `HBaseSnapshotRowInputFormat` and `TableSnapshotInputFormat`*
- *Extract 2 interfaces including `HBaseTableScannerAware` and `ResultToTupleMapper`*
- *Add `HBaseSnapshotReadExample`*
## Verifying this change
This change is already covered by existing tests as follows, and new test cases has been added as well.
`org.apache.flink.addons.hbase.HBaseConnectorITCase`
This change added tests and can be verified as follows:
- *Manually create one snapshot for a specific HBase table, and use TableSnapshotInputFormat to do full scan.*
- *Running existing HBaseReadExample to do full scan.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (not applicable / **docs** / **JavaDocs** / not documented)
- For document, please visit [JIRA ticket](https://issues.apache.org/jira/projects/FLINK/issues/FLINK-8862?filter=allopenissues), a detailed design doc and class diagram have been attached.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/neoremind/flink snapshot
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5639.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5639
----
commit 0b36b434f987a971b6463ce3441c483380cfa9dd
Author: neoremind <xu...@...>
Date: 2018-03-05T14:14:09Z
Support HBase snapshot read
----
---
[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Posted by neoremind <gi...@git.apache.org>.
Github user neoremind commented on the issue:
https://github.com/apache/flink/pull/5639
@zentol I was thinking could you help me review the feature and code? Or is there anyone more appropriate for this PR? many thanks.
---
[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Posted by neoremind <gi...@git.apache.org>.
Github user neoremind commented on the issue:
https://github.com/apache/flink/pull/5639
@ramkrish86 @fhueske could you help to review this PR? Since I noticed that you guys contribute most of the code. This update enables HBase snapshot read and I refactor some of the code and test cases , you can find design doc and class diagram on https://issues.apache.org/jira/projects/FLINK/issues/FLINK-8862?filter=allopenissues. Thanks!
---
[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Posted by neoremind <gi...@git.apache.org>.
Github user neoremind commented on the issue:
https://github.com/apache/flink/pull/5639
@fhueske Thanks for your response. I understand this case. Please take your time, hope this PR can be reviewed in the future and help people who needed. Thanks!
---
[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/5639
Thanks for the PR @neoremind.
At the moment, the community is busy working on the 1.5 release which means that PRs for 1.5 fixes have priority right now. Also a large contribution such as this one takes a lot of time to review. Unfortunately, I won't be able to review the PR in the near future. Best, Fabian
---