kafka streams, docker/k18s and rocksdb - storage performance

["Marasoiu, Nicu" <ni...@metrosystems.net>]

Hi,
When using kafka streams & stateful transformations (rocksdb) and docker and Kubernetes, what are the concerns for storage - I mean, I know that writing to disk in Docker into the container is less performant than mounting a direct volume. However, in our setup a separate team is handling, and theoretically we should be stateless in the microservices nodes, .. I am just wondering what the performance penalty would be if the writes would be done without any special concern,
Thank you,
Nicu
Geschäftsanschrift/Business address: METRO SYSTEMS GmbH, Metro-Straße 12, 40235 Düsseldorf, Germany
Aufsichtsrat/Supervisory Board: Heiko Hutmacher (Vorsitzender/ Chairman)
Geschäftsführung/Management Board: Dr. Dirk Toepfer (Vorsitzender/CEO), Wim van Herwijnen
Sitz Düsseldorf, Amtsgericht Düsseldorf, HRB 18232/Registered Office Düsseldorf, Commercial Register of the Düsseldorf Local Court, HRB 18232

Betreffend Mails von *@metrosystems.net
Die in dieser E-Mail enthaltenen Nachrichten und Anhänge sind ausschließlich für den bezeichneten Adressaten bestimmt. Sie können rechtlich geschützte, vertrauliche Informationen enthalten. Falls Sie nicht der bezeichnete Empfänger oder zum Empfang dieser E-Mail nicht berechtigt sind, ist die Verwendung, Vervielfältigung oder Weitergabe der Nachrichten und Anhänge untersagt. Falls Sie diese E-Mail irrtümlich erhalten haben, informieren Sie bitte unverzüglich den Absender und vernichten Sie die E-Mail.

Regarding mails from *@metrosystems.net
This e-mail message and any attachment are intended exclusively for the named addressee. They may contain confidential information which may also be protected by professional secrecy. Unless you are the named addressee (or authorised to receive for the addressee) you may not copy or use this message or any attachment or disclose the contents to anyone else. If this e-mail was sent to you by mistake please notify the sender immediately and delete this e-mail.

Re: kafka streams, docker/k18s and rocksdb - storage performance

[Pegerto Fernandez Torres <pe...@opencredo.com>]

Hi Nicu,

I think the best you can do is start with an empty dir, that will give you
access to a not layered filesystem, but you need to take in consideration
that you will stream data every time that the container restart, so a
rolling update can be slow.

My suggestion is to check with you platform team, I know them and I am sure
they are happy to help, and understand if the restoration time for the data
size you plan is acceptable.

Regards.

On 28 February 2018 at 08:39, Marasoiu, Nicu <nicu.marasoiu@metrosystems.net
> wrote:

> Hi,
> When using kafka streams & stateful transformations (rocksdb) and docker
> and Kubernetes, what are the concerns for storage - I mean, I know that
> writing to disk in Docker into the container is less performant than
> mounting a direct volume. However, in our setup a separate team is
> handling, and theoretically we should be stateless in the microservices
> nodes, .. I am just wondering what the performance penalty would be if the
> writes would be done without any special concern,
> Thank you,
> Nicu
> Geschäftsanschrift/Business address: METRO SYSTEMS GmbH, Metro-Straße 12,
> 40235 Düsseldorf, Germany
> Aufsichtsrat/Supervisory Board: Heiko Hutmacher (Vorsitzender/ Chairman)
> Geschäftsführung/Management Board: Dr. Dirk Toepfer (Vorsitzender/CEO),
> Wim van Herwijnen
> Sitz Düsseldorf, Amtsgericht Düsseldorf, HRB 18232/Registered Office
> Düsseldorf, Commercial Register of the Düsseldorf Local Court, HRB 18232
>
> Betreffend Mails von *@metrosystems.net
> Die in dieser E-Mail enthaltenen Nachrichten und Anhänge sind
> ausschließlich für den bezeichneten Adressaten bestimmt. Sie können
> rechtlich geschützte, vertrauliche Informationen enthalten. Falls Sie nicht
> der bezeichnete Empfänger oder zum Empfang dieser E-Mail nicht berechtigt
> sind, ist die Verwendung, Vervielfältigung oder Weitergabe der Nachrichten
> und Anhänge untersagt. Falls Sie diese E-Mail irrtümlich erhalten haben,
> informieren Sie bitte unverzüglich den Absender und vernichten Sie die
> E-Mail.
>
> Regarding mails from *@metrosystems.net
> This e-mail message and any attachment are intended exclusively for the
> named addressee. They may contain confidential information which may also
> be protected by professional secrecy. Unless you are the named addressee
> (or authorised to receive for the addressee) you may not copy or use this
> message or any attachment or disclose the contents to anyone else. If this
> e-mail was sent to you by mistake please notify the sender immediately and
> delete this e-mail.
>
>


-- 

Pegerto Fernandez

*Lead Consultant *

-- 


Check out our new blog by Miles Wilson, "Fargate As An Enabler For 
Serverless Continuous Delivery" <https://goo.gl/2TCHtr>


opencredo.com <http://www.opencredo.com/> . Twitter 
<https://twitter.com/OpenCredo> . LinkedIn 
<https://www.linkedin.com/company/opencredo>


OpenCredo Ltd -- Delivering Software Innovation

Registered Office:  5-11 Lavington Street, London, SE1 0NZ
Registered in UK. No 3943999