You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sudan S <su...@cred.club> on 2020/06/22 13:22:00 UTC
Rocksdb state directory path in EMR
Hi,
I have enabled rocksdb(State store) with s3 (external checkpoint) on EMR .
I am using rocksdb as state store with ValueState and checkpoints are
stored in s3.
I am able to see checkpoints in s3 and functionality with respect to state
store is working fine.
But i am trying to dissect rocksdb folder structure but i am not able to
find them on EMR. I am checking /mnt/yarn folder.
Cn u plz help on where flink persists store on EMR when rocksdb is used as
state backend?
--
*"The information contained in this e-mail and any accompanying documents
may contain information that is confidential or otherwise protected from
disclosure. If you are not the intended recipient of this message, or if
this message has been addressed to you in error, please immediately alert
the sender by replying to this e-mail and then delete this message,
including any attachments. Any dissemination, distribution or other use of
the contents of this message by anyone other than the intended recipient is
strictly prohibited. All messages sent to and from this e-mail address may
be monitored as permitted by applicable law and regulations to ensure
compliance with our internal policies and to protect our business."*
Re: Rocksdb state directory path in EMR
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi,
If I understand you correctly, you want to check the local RocksDB
files, right? They are stored locally on each TaskManager in a temporary
directory.
This can be configured via "state.backend.rocksdb.localdir"[1]. If not
specified it will use the globally defined temporary directory set via
"io.tmp.dirs"[2], which defaults to : "'LOCAL_DIRS' on Yarn.
'_FLINK_TMP_DIR' on Mesos. System.getProperty("java.io.tmpdir") in
standalone."
Best,
Dawid
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#state-backend-rocksdb-localdir
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#io-tmp-dirs
On 22/06/2020 15:22, Sudan S wrote:
> Hi,
>
> I have enabled rocksdb(State store) with s3 (external checkpoint) on EMR .
>
> I am using rocksdb as state store with ValueState and checkpoints are
> stored in s3.
>
> I am able to see checkpoints in s3 and functionality with respect to
> state store is working fine.
>
> But i am trying to dissect rocksdb folder structure but i am not able
> to find them on EMR. I am checking /mnt/yarn folder.
>
> Cn u plz help on where flink persists store on EMR when rocksdb is
> used as state backend?
>
>
>
>
>
> ------------------------------------------------------------------------
> /"The information contained in this e-mail and any accompanying
> documents may contain information that is confidential or otherwise
> protected from disclosure. If you are not the intended recipient of
> this message, or if this message has been addressed to you in error,
> please immediately alert the sender by replying to this e-mail and
> then delete this message, including any attachments. Any
> dissemination, distribution or other use of the contents of this
> message by anyone other than the intended recipient is strictly
> prohibited. All messages sent to and from this e-mail address may be
> monitored as permitted by applicable law and regulations to ensure
> compliance with our internal policies and to protect our business."/
> ------------------------------------------------------------------------