You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Mike Thomsen <mi...@gmail.com> on 2018/02/25 20:42:59 UTC

Incrementally load HFiles outside of MR/Spark

I'm looking into creating HFiles directly from NiFi using the HBase API. It
seems pretty straight forward:

1. Open a HFile.Writer pointing to a file path in HDFS.
2. Write the cells with the HFile API.
3. Call the incremental loader API to have it tell HBase to load the
generated segments.

Is that right? If so, are there any gotchas that I should be aware of?

Thanks,

Mike

Re: Incrementally load HFiles outside of MR/Spark

Posted by Ted Yu <yu...@gmail.com>.

You can refer to HFilePerformanceEvaluation where creation of Writer is
demonstrated:

      writer = HFile.getWriterFactoryNoCache(conf)

          .withPath(fs, mf)

          .withFileContext(hFileContext)

          .withComparator(CellComparator.getInstance())

          .create();

Cheers

On Sun, Feb 25, 2018 at 12:42 PM, Mike Thomsen <mi...@gmail.com>
wrote:

> I'm looking into creating HFiles directly from NiFi using the HBase API. It
> seems pretty straight forward:
>
> 1. Open a HFile.Writer pointing to a file path in HDFS.
> 2. Write the cells with the HFile API.
> 3. Call the incremental loader API to have it tell HBase to load the
> generated segments.
>
> Is that right? If so, are there any gotchas that I should be aware of?
>
> Thanks,
>
> Mike
>