DirectMemory buffers

[Steven Yates <sy...@stevendyates.com>]

Hi devs, i am looking to get as much performance out of Samza as possible and am interested in looking at what effect a direct memory approach has on performance an whether frameworks like Kafka can be swapped out for a more direct off heap approach I am trialling this implementation now in my local env however I don't have exact metrics yet. I was wondering if you guys had any further thoughts on this?

-Steve

Re: DirectMemory buffers

[Steven Yates <sy...@stevendyates.com>]

Hey Chris, I would be very interested too? :) I am slowly working my way through it to be honest. The application I am/have been applying to Samza the whole time is a finance stock ticker application but I want it to absolutely hum. I am interested in looking at trying to exploit the sun.misc.Unsafe api to see if we can gain some performance improvements with off-heap data maps etc. The guarantees that Samza currently provides is something I haven't looked into yet but will certainly want to look into in the near future.

Thanks
Steve 


Excerpts from Chris Riccomini's message of 2014-09-17 01:17:03 +1000:
> Hey Steve,
> 
> I'd be very interested in hearing what you discover.
> 
> Most performance-related knowledge that I have is about tuning Kafka to go
> fast. :)
> 
> As far as implementation goes, I think you'll need to implement a
> SystemConsumer, SystemProducer, SystemAdmin, and SystemFactory in order to
> fully support direct memory. The main problem with "swapping" out Kafka is
> that you're going to lose some of Samza's guarantees. Samza depends a lot
> on the guarantees of the underlying streaming system for things like:
> 
> * Message ordering.
> * At-least once messaging.
> * Replayability (offsets).
> * Fault tolerance (replication).
> 
> If your direct memory implementation doesn¹t provide some of these
> features, then neither can Samza. That may be fine, or that may be
> unsatisfactory for your use case. Samza will work without these features,
> but makes no effort to provide them itself. This means if, for example,
> your direct memory implementation isn't repayable, then your offset
> checkpoints are useless in Samza, and will be disregarded (you'll always
> start consuming from wherever the direct memory SystemConsumer
> implementation decides to start).
> 
> Cheers,
> Chris
> 
> On 9/16/14 4:21 AM, "Steven Yates" <sy...@stevendyates.com> wrote:
> 
> >Hi devs, i am looking to get as much performance out of Samza as possible
> >and am interested in looking at what effect a direct memory approach has
> >on performance an whether frameworks like Kafka can be swapped out for a
> >more direct off heap approach I am trialling this implementation now in
> >my local env however I don't have exact metrics yet. I was wondering if
> >you guys had any further thoughts on this?
> >
> >-Steve

Re: DirectMemory buffers

[Chris Riccomini <cr...@linkedin.com.INVALID>]

Hey Steve,

I'd be very interested in hearing what you discover.

Most performance-related knowledge that I have is about tuning Kafka to go
fast. :)

As far as implementation goes, I think you'll need to implement a
SystemConsumer, SystemProducer, SystemAdmin, and SystemFactory in order to
fully support direct memory. The main problem with "swapping" out Kafka is
that you're going to lose some of Samza's guarantees. Samza depends a lot
on the guarantees of the underlying streaming system for things like:

* Message ordering.
* At-least once messaging.
* Replayability (offsets).
* Fault tolerance (replication).

If your direct memory implementation doesn¹t provide some of these
features, then neither can Samza. That may be fine, or that may be
unsatisfactory for your use case. Samza will work without these features,
but makes no effort to provide them itself. This means if, for example,
your direct memory implementation isn't repayable, then your offset
checkpoints are useless in Samza, and will be disregarded (you'll always
start consuming from wherever the direct memory SystemConsumer
implementation decides to start).

Cheers,
Chris

On 9/16/14 4:21 AM, "Steven Yates" <sy...@stevendyates.com> wrote:

>Hi devs, i am looking to get as much performance out of Samza as possible
>and am interested in looking at what effect a direct memory approach has
>on performance an whether frameworks like Kafka can be swapped out for a
>more direct off heap approach I am trialling this implementation now in
>my local env however I don't have exact metrics yet. I was wondering if
>you guys had any further thoughts on this?
>
>-Steve