Heartbeat message limit

[Dushyant Bansal <Du...@microsoft.com>]

Hi REEF team,

I'm in Azure Stream Analytics at Microsoft. We're adding data to the heartbeat that gets sent by the REEF task to the Driver.  I want to make sure we don't overflow any message buffers.  There is general guidance on the REEF site<http://reef.apache.org/glossary.html> to keep the message to keep the Driver performant at scale. But is there a limit on the message size beyond which the communication will break?  Is this limit configurable? Is there any guidance on how much load is considered "too much" load for the Driver?

"It is wise for that message to be small, as we otherwise run the risk of overwhelming the Driver with heartbeat traffic at scale."

Thanks,
Dushyant

Re: Heartbeat message limit

[Markus Weimer <ma...@weimo.de>]

Hi,

we don't enforce a limit on the heartbeat size. I would also be
surprised if would break down before a reasonable limit (don't expect
it to work with data greater than main memory). In practice, the
message size needs to be transmittable within the heartbeat interval
from all Evaluators.

That means that the faster the heartbeat interval you set, the smaller
the message needs to be. Also, the more Evaluators you have, the
smaller your messages need to be. Note that all heartbeats have to go
through the Driver's network interface. So its speed will be the upper
bound for all of this.

There are people on the list with much more background on this, but my
hunch is that messages below the 2kB range shouldn't be a problem in
realist applications (100s of Evaluators).

Markus

On Tue, May 10, 2016 at 11:45 AM, Dushyant Bansal
<Du...@microsoft.com> wrote:
> Hi REEF team,
>
> I'm in Azure Stream Analytics at Microsoft. We're adding data to the heartbeat that gets sent by the REEF task to the Driver.  I want to make sure we don't overflow any message buffers.  There is general guidance on the REEF site<http://reef.apache.org/glossary.html> to keep the message to keep the Driver performant at scale. But is there a limit on the message size beyond which the communication will break?  Is this limit configurable? Is there any guidance on how much load is considered "too much" load for the Driver?
>
> "It is wise for that message to be small, as we otherwise run the risk of overwhelming the Driver with heartbeat traffic at scale."
>
> Thanks,
> Dushyant