You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Siew Wai Yow <wa...@hotmail.com> on 2018/07/06 13:58:41 UTC
flink rocksdb where to configure mount point
Hi,
We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.
RocksDBStateBackend rocksdb = new RocksDBStateBackend(new FsStateBackend(hdfs://), true);
env.setStateBackend(rocksdb);
Questions,
1. What happen if large state is larger than the tmpfs memory size? Will flink do something?
2. Is it possible to configure rocksdb dir via flink?
Thank you.
Regards,
Yow
Re: flink rocksdb where to configure mount point
Posted by Siew Wai Yow <wa...@hotmail.com>.
Hi Yun Tang,
Thanks to your reply, this is exactly the answer we need.
also thanks @CongXian's reply.
-Siew Wai
________________________________
From: Tang Cloud <my...@live.com>
Sent: Monday, July 9, 2018 11:05 AM
To: Siew Wai Yow; user@flink.apache.org
Subject: 答复: flink rocksdb where to configure mount point
Hi Siew,
As far as I know, if the state size is larger than the tmpfs memory size, which means tmpfs will be occupied fully leading to "IOException: No space left on device". However, you just need to call RocksDBStateBackend#setDbStoragePaths method to re-configure the local DB storage dir to a larger disk volume and don't worry about the data completeness as the doc of RocksDBStateBackend#getDbStoragePaths method said:
"Under these directories on the TaskManager, RocksDB stores its SST files and metadata files. These directories do not need to be persistent, they can be ephermeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints."
Regards,
Yun Tang
________________________________
发件人: Siew Wai Yow <wa...@hotmail.com>
发送时间: 2018年7月6日 13:58
收件人: user@flink.apache.org
主题: flink rocksdb where to configure mount point
Hi,
We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.
RocksDBStateBackend rocksdb = new RocksDBStateBackend(new FsStateBackend(hdfs://), true);
env.setStateBackend(rocksdb);
Questions,
1. What happen if large state is larger than the tmpfs memory size? Will flink do something?
2. Is it possible to configure rocksdb dir via flink?
Thank you.
Regards,
Yow
答复: flink rocksdb where to configure mount point
Posted by Tang Cloud <my...@live.com>.
Hi Siew,
As far as I know, if the state size is larger than the tmpfs memory size, which means tmpfs will be occupied fully leading to "IOException: No space left on device". However, you just need to call RocksDBStateBackend#setDbStoragePaths method to re-configure the local DB storage dir to a larger disk volume and don't worry about the data completeness as the doc of RocksDBStateBackend#getDbStoragePaths method said:
"Under these directories on the TaskManager, RocksDB stores its SST files and metadata files. These directories do not need to be persistent, they can be ephermeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints."
Regards,
Yun Tang
________________________________
发件人: Siew Wai Yow <wa...@hotmail.com>
发送时间: 2018年7月6日 13:58
收件人: user@flink.apache.org
主题: flink rocksdb where to configure mount point
Hi,
We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.
RocksDBStateBackend rocksdb = new RocksDBStateBackend(new FsStateBackend(hdfs://), true);
env.setStateBackend(rocksdb);
Questions,
1. What happen if large state is larger than the tmpfs memory size? Will flink do something?
2. Is it possible to configure rocksdb dir via flink?
Thank you.
Regards,
Yow