You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by LakeShen <sh...@gmail.com> on 2020/05/06 04:06:48 UTC
What is the RocksDB local directory in flink checkpointing?
Hi community,
Now I have a question about flink checkpoint local directory , our flink
version is 1.6, job mode is
flink on yarn per job . I saw the flink source code , and I find the flink
checkpoint local directory is
/tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
into the /tmp dir ,I
couldn't find the flink checkpoint state local directory.
What is the RocksDB local directory in flink checkpointing? I am looking
forward to your reply.
Best,
LakeShen
Re: What is the RocksDB local directory in flink checkpointing?
Posted by Yun Tang <my...@live.com>.
Hi LakeShen
You could refer to [1] and [2] to know the temporary directory in YARN, the related log could be
"Setting directories for temporary files to: " or "Overriding Fink's temporary file directories with those specified in the Flink config: "
[1] https://github.com/apache/flink/blob/0dda6fe9dff4f667b110cda39bfe9738ba615b24/flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskExecutorRunner.java#L103
[2] https://github.com/apache/flink/blob/0dda6fe9dff4f667b110cda39bfe9738ba615b24/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L478-L489
Best
Yun Tang
________________________________
From: Till Rohrmann <tr...@apache.org>
Sent: Wednesday, May 6, 2020 17:35
To: LakeShen <sh...@gmail.com>
Cc: dev <de...@flink.apache.org>; user <us...@flink.apache.org>; user-zh <us...@flink.apache.org>
Subject: Re: What is the RocksDB local directory in flink checkpointing?
Hi LakeShen,
`state.backend.rocksdb.localdir` defines the directory in which RocksDB
will store its local files. Local files are RocksDB's SST and metadata
files for example. This directory does not need to be persisted. If the
config option is not configured, then it will use the nodes temporary file
directory.
Cheers,
Till
On Wed, May 6, 2020 at 6:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi community,
>
> Now I have a question about flink checkpoint local directory , our flink
> version is 1.6, job mode is
>
> flink on yarn per job . I saw the flink source code , and I find the flink
> checkpoint local directory is
>
> /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
> into the /tmp dir ,I
>
> couldn't find the flink checkpoint state local directory.
>
> What is the RocksDB local directory in flink checkpointing? I am looking
> forward to your reply.
>
> Best,
> LakeShen
>
Re: What is the RocksDB local directory in flink checkpointing?
Posted by Yun Tang <my...@live.com>.
Hi LakeShen
You could refer to [1] and [2] to know the temporary directory in YARN, the related log could be
"Setting directories for temporary files to: " or "Overriding Fink's temporary file directories with those specified in the Flink config: "
[1] https://github.com/apache/flink/blob/0dda6fe9dff4f667b110cda39bfe9738ba615b24/flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskExecutorRunner.java#L103
[2] https://github.com/apache/flink/blob/0dda6fe9dff4f667b110cda39bfe9738ba615b24/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L478-L489
Best
Yun Tang
________________________________
From: Till Rohrmann <tr...@apache.org>
Sent: Wednesday, May 6, 2020 17:35
To: LakeShen <sh...@gmail.com>
Cc: dev <de...@flink.apache.org>; user <us...@flink.apache.org>; user-zh <us...@flink.apache.org>
Subject: Re: What is the RocksDB local directory in flink checkpointing?
Hi LakeShen,
`state.backend.rocksdb.localdir` defines the directory in which RocksDB
will store its local files. Local files are RocksDB's SST and metadata
files for example. This directory does not need to be persisted. If the
config option is not configured, then it will use the nodes temporary file
directory.
Cheers,
Till
On Wed, May 6, 2020 at 6:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi community,
>
> Now I have a question about flink checkpoint local directory , our flink
> version is 1.6, job mode is
>
> flink on yarn per job . I saw the flink source code , and I find the flink
> checkpoint local directory is
>
> /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
> into the /tmp dir ,I
>
> couldn't find the flink checkpoint state local directory.
>
> What is the RocksDB local directory in flink checkpointing? I am looking
> forward to your reply.
>
> Best,
> LakeShen
>
Re: What is the RocksDB local directory in flink checkpointing?
Posted by Yun Tang <my...@live.com>.
Hi LakeShen
You could refer to [1] and [2] to know the temporary directory in YARN, the related log could be
"Setting directories for temporary files to: " or "Overriding Fink's temporary file directories with those specified in the Flink config: "
[1] https://github.com/apache/flink/blob/0dda6fe9dff4f667b110cda39bfe9738ba615b24/flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskExecutorRunner.java#L103
[2] https://github.com/apache/flink/blob/0dda6fe9dff4f667b110cda39bfe9738ba615b24/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L478-L489
Best
Yun Tang
________________________________
From: Till Rohrmann <tr...@apache.org>
Sent: Wednesday, May 6, 2020 17:35
To: LakeShen <sh...@gmail.com>
Cc: dev <de...@flink.apache.org>; user <us...@flink.apache.org>; user-zh <us...@flink.apache.org>
Subject: Re: What is the RocksDB local directory in flink checkpointing?
Hi LakeShen,
`state.backend.rocksdb.localdir` defines the directory in which RocksDB
will store its local files. Local files are RocksDB's SST and metadata
files for example. This directory does not need to be persisted. If the
config option is not configured, then it will use the nodes temporary file
directory.
Cheers,
Till
On Wed, May 6, 2020 at 6:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi community,
>
> Now I have a question about flink checkpoint local directory , our flink
> version is 1.6, job mode is
>
> flink on yarn per job . I saw the flink source code , and I find the flink
> checkpoint local directory is
>
> /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
> into the /tmp dir ,I
>
> couldn't find the flink checkpoint state local directory.
>
> What is the RocksDB local directory in flink checkpointing? I am looking
> forward to your reply.
>
> Best,
> LakeShen
>
Re: What is the RocksDB local directory in flink checkpointing?
Posted by Till Rohrmann <tr...@apache.org>.
Hi LakeShen,
`state.backend.rocksdb.localdir` defines the directory in which RocksDB
will store its local files. Local files are RocksDB's SST and metadata
files for example. This directory does not need to be persisted. If the
config option is not configured, then it will use the nodes temporary file
directory.
Cheers,
Till
On Wed, May 6, 2020 at 6:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi community,
>
> Now I have a question about flink checkpoint local directory , our flink
> version is 1.6, job mode is
>
> flink on yarn per job . I saw the flink source code , and I find the flink
> checkpoint local directory is
>
> /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
> into the /tmp dir ,I
>
> couldn't find the flink checkpoint state local directory.
>
> What is the RocksDB local directory in flink checkpointing? I am looking
> forward to your reply.
>
> Best,
> LakeShen
>
Re: What is the RocksDB local directory in flink checkpointing?
Posted by Till Rohrmann <tr...@apache.org>.
Hi LakeShen,
`state.backend.rocksdb.localdir` defines the directory in which RocksDB
will store its local files. Local files are RocksDB's SST and metadata
files for example. This directory does not need to be persisted. If the
config option is not configured, then it will use the nodes temporary file
directory.
Cheers,
Till
On Wed, May 6, 2020 at 6:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi community,
>
> Now I have a question about flink checkpoint local directory , our flink
> version is 1.6, job mode is
>
> flink on yarn per job . I saw the flink source code , and I find the flink
> checkpoint local directory is
>
> /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
> into the /tmp dir ,I
>
> couldn't find the flink checkpoint state local directory.
>
> What is the RocksDB local directory in flink checkpointing? I am looking
> forward to your reply.
>
> Best,
> LakeShen
>
回复：What is the RocksDB local directory in flink checkpointing?
Posted by 范瑞 <83...@qq.com>.
Hi LakeShen,
Default use yarn's tmp dir.
Best,
fanrui
------------------ 原始邮件 ------------------
发件人: "LakeShen"<shenleifighting@gmail.com>;
发送时间: 2020年5月6日(星期三) 中午12:06
收件人: "dev"<dev@flink.apache.org>;"user"<user@flink.apache.org>;"user-zh"<user-zh@flink.apache.org>;
主题: What is the RocksDB local directory in flink checkpointing?
Hi community,
Now I have a question about flink checkpoint local directory , our flink
version is 1.6, job mode is
flink on yarn per job . I saw the flink source code , and I find the flink
checkpoint local directory is
/tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
into the /tmp dir ,I
couldn't find the flink checkpoint state local directory.
What is the RocksDB local directory in flink checkpointing? I am looking
forward to your reply.
Best,
LakeShen
Re: What is the RocksDB local directory in flink checkpointing?
Posted by Till Rohrmann <tr...@apache.org>.
Hi LakeShen,
`state.backend.rocksdb.localdir` defines the directory in which RocksDB
will store its local files. Local files are RocksDB's SST and metadata
files for example. This directory does not need to be persisted. If the
config option is not configured, then it will use the nodes temporary file
directory.
Cheers,
Till
On Wed, May 6, 2020 at 6:07 AM LakeShen <sh...@gmail.com> wrote:
> Hi community,
>
> Now I have a question about flink checkpoint local directory , our flink
> version is 1.6, job mode is
>
> flink on yarn per job . I saw the flink source code , and I find the flink
> checkpoint local directory is
>
> /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go
> into the /tmp dir ,I
>
> couldn't find the flink checkpoint state local directory.
>
> What is the RocksDB local directory in flink checkpointing? I am looking
> forward to your reply.
>
> Best,
> LakeShen
>