You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jayant Ameta <wi...@gmail.com> on 2018/03/05 11:57:04 UTC
Rocksdb in production
Hi,
I wanted to know how's the online support and resources for RocksDB? I want
to use RocksDB as the state backend, but I'm not sure how active the
community is. Can anyone vouch for it?
Re: Rocksdb in production
Posted by Fabian Hueske <fh...@gmail.com>.
That depends on your job and the setup.
Remember that all operators will write their checkpoint data into that file
system.
If the state grows very large and only have an NFS with little write
performance, it might be a problem. But the same would apply to HDFS as
well.
2018-03-06 2:51 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
> Thanks Fabian.
> Will there be any performance issues if I use NFS as the shared filesystem
> (as compared to HDFS or S3)?
>
> Jayant Ameta
>
> On Mon, Mar 5, 2018 at 10:31 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Yes, that is correct.
>>
>> 2018-03-05 8:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
>>
>>> Oh! Somehow I missed while reading the documentation that RocksDB is
>>> embedded in Flink.
>>>
>>> Also, irrespective of state backend being filesystem or rocksdb, I'll
>>> have to setup a shared filesystem (HDFS, S3, etc). Is my understanding
>>> correct?
>>>
>>>
>>> Jayant Ameta
>>>
>>> On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> RockDB is an embedded key-value storage that is internally used by
>>>> Flink. There is no need to setup a RocksDB database or service yourself.
>>>> All of that is done by Flink.
>>>> As a Flink user that uses the RockDB state backend, you won't get in
>>>> touch with RocksDB itself.
>>>>
>>>> Besides that, RocksDB is developed by Facebook [1] and is a fairly
>>>> active project.
>>>>
>>>> Best, Fabian
>>>>
>>>> [1] https://github.com/facebook/rocksdb
>>>>
>>>> 2018-03-05 3:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
>>>>
>>>>> Hi,
>>>>> I wanted to know how's the online support and resources for RocksDB? I
>>>>> want to use RocksDB as the state backend, but I'm not sure how active the
>>>>> community is. Can anyone vouch for it?
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Rocksdb in production
Posted by Jayant Ameta <wi...@gmail.com>.
Thanks Fabian.
Will there be any performance issues if I use NFS as the shared filesystem
(as compared to HDFS or S3)?
Jayant Ameta
On Mon, Mar 5, 2018 at 10:31 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Yes, that is correct.
>
> 2018-03-05 8:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
>
>> Oh! Somehow I missed while reading the documentation that RocksDB is
>> embedded in Flink.
>>
>> Also, irrespective of state backend being filesystem or rocksdb, I'll
>> have to setup a shared filesystem (HDFS, S3, etc). Is my understanding
>> correct?
>>
>>
>> Jayant Ameta
>>
>> On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
>>
>>> Hi,
>>> RockDB is an embedded key-value storage that is internally used by
>>> Flink. There is no need to setup a RocksDB database or service yourself.
>>> All of that is done by Flink.
>>> As a Flink user that uses the RockDB state backend, you won't get in
>>> touch with RocksDB itself.
>>>
>>> Besides that, RocksDB is developed by Facebook [1] and is a fairly
>>> active project.
>>>
>>> Best, Fabian
>>>
>>> [1] https://github.com/facebook/rocksdb
>>>
>>> 2018-03-05 3:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
>>>
>>>> Hi,
>>>> I wanted to know how's the online support and resources for RocksDB? I
>>>> want to use RocksDB as the state backend, but I'm not sure how active the
>>>> community is. Can anyone vouch for it?
>>>>
>>>
>>>
>>
>
Re: Rocksdb in production
Posted by Fabian Hueske <fh...@gmail.com>.
Yes, that is correct.
2018-03-05 8:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
> Oh! Somehow I missed while reading the documentation that RocksDB is
> embedded in Flink.
>
> Also, irrespective of state backend being filesystem or rocksdb, I'll have
> to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct?
>
>
> Jayant Ameta
>
> On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi,
>> RockDB is an embedded key-value storage that is internally used by Flink.
>> There is no need to setup a RocksDB database or service yourself. All of
>> that is done by Flink.
>> As a Flink user that uses the RockDB state backend, you won't get in
>> touch with RocksDB itself.
>>
>> Besides that, RocksDB is developed by Facebook [1] and is a fairly active
>> project.
>>
>> Best, Fabian
>>
>> [1] https://github.com/facebook/rocksdb
>>
>> 2018-03-05 3:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
>>
>>> Hi,
>>> I wanted to know how's the online support and resources for RocksDB? I
>>> want to use RocksDB as the state backend, but I'm not sure how active the
>>> community is. Can anyone vouch for it?
>>>
>>
>>
>
Re: Rocksdb in production
Posted by Jayant Ameta <wi...@gmail.com>.
Oh! Somehow I missed while reading the documentation that RocksDB is
embedded in Flink.
Also, irrespective of state backend being filesystem or rocksdb, I'll have
to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct?
Jayant Ameta
On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi,
> RockDB is an embedded key-value storage that is internally used by Flink.
> There is no need to setup a RocksDB database or service yourself. All of
> that is done by Flink.
> As a Flink user that uses the RockDB state backend, you won't get in touch
> with RocksDB itself.
>
> Besides that, RocksDB is developed by Facebook [1] and is a fairly active
> project.
>
> Best, Fabian
>
> [1] https://github.com/facebook/rocksdb
>
> 2018-03-05 3:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
>
>> Hi,
>> I wanted to know how's the online support and resources for RocksDB? I
>> want to use RocksDB as the state backend, but I'm not sure how active the
>> community is. Can anyone vouch for it?
>>
>
>
Re: Rocksdb in production
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
RockDB is an embedded key-value storage that is internally used by Flink.
There is no need to setup a RocksDB database or service yourself. All of
that is done by Flink.
As a Flink user that uses the RockDB state backend, you won't get in touch
with RocksDB itself.
Besides that, RocksDB is developed by Facebook [1] and is a fairly active
project.
Best, Fabian
[1] https://github.com/facebook/rocksdb
2018-03-05 3:57 GMT-08:00 Jayant Ameta <wi...@gmail.com>:
> Hi,
> I wanted to know how's the online support and resources for RocksDB? I
> want to use RocksDB as the state backend, but I'm not sure how active the
> community is. Can anyone vouch for it?
>