You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Torok, David" <Da...@comcast.com> on 2016/12/23 18:20:43 UTC
RocksDB Windows / Flink 1.1.4 - requires Hadoop?
Hi,
I tried to use the newly-supported RocksDB backend (Flink 1.1.4) on my Windows laptop.
However, it is not creating any state and is throwing NPE while trying to call:
org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(HDFSCopyFromLocal.java:47)
which eventually gets down to not finding Hadoop Winutils.
I am not using an hdfs path....
String statepath = parameterTool.get("statepath", "file:///c:/temp/flink/checkpoints<file:///c:\temp\flink\checkpoints>");
env.setStateBackend(new RocksDBStateBackend(statepath));
I notice that in v1.2.0 the async checkpoint mode has been removed
(https://issues.apache.org/jira/browse/FLINK-4340)
In the interim is there any hint to using RocksDB on Windows, or should I just revert to File for my playground?
Cheers
Dave Torok
Re: RocksDB Windows / Flink 1.1.4 - requires Hadoop?
Posted by Stephan Ewen <se...@apache.org>.
Hi Dave!
In 1.1.x, the default mode of RocksDB uses some Hadoop utilities,
regardless of the filesystem used. I think that was a design mistake and we
rectified that in the upcoming 1.2.x
For 1.1.4, I would use "enableFullyAsyncSnapchots()" on the RocksDB state
backend - that mode should also not depend on Hadoop utilities.
Best,
Stephan
On Fri, Dec 23, 2016 at 7:20 PM, Torok, David <Da...@comcast.com>
wrote:
> Hi,
>
>
>
> I tried to use the newly-supported RocksDB backend (Flink 1.1.4) on my
> Windows laptop.
>
>
>
> However, it is not creating any state and is throwing NPE while trying to
> call:
>
>
>
> org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(
> HDFSCopyFromLocal.java:47)
>
>
>
> which eventually gets down to not finding Hadoop Winutils.
>
>
>
> I am not using an hdfs path....
>
>
>
> String statepath = parameterTool.get("statepath", "file:///c:/temp/flink/
> checkpoints");
>
> env.setStateBackend(new RocksDBStateBackend(statepath));
>
>
>
> I notice that in v1.2.0 the async checkpoint mode has been removed
>
> (https://issues.apache.org/jira/browse/FLINK-4340)
>
>
>
> In the interim is there any hint to using RocksDB on Windows, or should I
> just revert to File for my playground?
>
>
>
> Cheers
>
> Dave Torok
>
>
>