You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by neoremind <gi...@git.apache.org> on 2018/03/05 14:29:52 UTC

[GitHub] flink pull request #5639: [FLINK-8862] [HBase] Support HBase snapshot read

GitHub user neoremind opened a pull request:

    https://github.com/apache/flink/pull/5639

    [FLINK-8862] [HBase] Support HBase snapshot read

    ## What is the purpose of the change
    
    *Flink-hbase connector only supports reading/scanning HBase over region server scanner, there is also [snapshot](http://hbase.apache.org/book.html#ops.snapshots) scanning solution, just like Hadoop provides 2 ways to scan HBase, one is [TableInputFormat](https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableInputFormat.html), the other is [TableSnapshotInputFormat](https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.html), so it would be great if flink supports both solutions to ensure more wider usage scope and provide alternatives for users.*
    
    
    ## Brief change log
    
      - *Create `TableInputSplitStrategy` interface and its implementations as abstraction logic for `AbstractTableInputFormat`*
      - *Update `HBaseRowInputFormat` and `TableInputFormat`*
      - *Add `HBaseSnapshotRowInputFormat` and `TableSnapshotInputFormat`*
      - *Extract 2 interfaces including `HBaseTableScannerAware` and `ResultToTupleMapper`*
      - *Add `HBaseSnapshotReadExample`*
    
    
    ## Verifying this change
    
    This change is already covered by existing tests as follows, and new test cases has been added as well.
    
    `org.apache.flink.addons.hbase.HBaseConnectorITCase`
    
    This change added tests and can be verified as follows:
    
      - *Manually create one snapshot for a specific HBase table, and use TableSnapshotInputFormat to do full scan.*
      - *Running existing HBaseReadExample to do full scan.*
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (yes / **no**)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
      - The serializers: (yes / **no** / don't know)
      - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
      - The S3 file system connector: (yes / **no** / don't know)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (**yes** / no)
      - If yes, how is the feature documented? (not applicable / **docs** / **JavaDocs** / not documented)
      - For document, please visit [JIRA ticket](https://issues.apache.org/jira/projects/FLINK/issues/FLINK-8862?filter=allopenissues), a detailed design doc and class diagram have been attached.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/neoremind/flink snapshot

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5639
    
----
commit 0b36b434f987a971b6463ce3441c483380cfa9dd
Author: neoremind <xu...@...>
Date:   2018-03-05T14:14:09Z

    Support HBase snapshot read

----


---

[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read

Posted by neoremind <gi...@git.apache.org>.
Github user neoremind commented on the issue:

    https://github.com/apache/flink/pull/5639
  
    @zentol I was thinking could you help me review the feature and code? Or is there anyone more appropriate for this PR? many thanks.


---

[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read

Posted by neoremind <gi...@git.apache.org>.
Github user neoremind commented on the issue:

    https://github.com/apache/flink/pull/5639
  
    @ramkrish86 @fhueske could you help to review this PR? Since I noticed that you guys contribute most of the code. This update enables HBase snapshot read and I refactor some of the code and test cases , you can find design doc and class diagram on https://issues.apache.org/jira/projects/FLINK/issues/FLINK-8862?filter=allopenissues. Thanks!


---

[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read

Posted by neoremind <gi...@git.apache.org>.
Github user neoremind commented on the issue:

    https://github.com/apache/flink/pull/5639
  
    @fhueske Thanks for your response. I understand this case. Please take your time, hope this PR can be reviewed in the future and help people who needed. Thanks!


---

[GitHub] flink issue #5639: [FLINK-8862] [HBase] Support HBase snapshot read

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the issue:

    https://github.com/apache/flink/pull/5639
  
    Thanks for the PR @neoremind.
    At the moment, the community is busy working on the 1.5 release which means that PRs for 1.5 fixes have priority right now. Also a large contribution such as this one takes a lot of time to review. Unfortunately, I won't be able to review the PR in the near future. Best, Fabian


---