spark streaming input rate strange

[patcharee <Pa...@uni.no>]

Hi,

I have a streaming application with
- 1 sec interval
- accept data from a simulation through MulticastSocket

The simulation sent out data using multiple clients/threads every 1 sec 
interval. The input rate accepted by the streaming looks strange.
- When clients = 10,000 the event rate raises up to 10,000, stays at 
10,000 a while and drops to about 7000-8000.
- When clients = 20,000 the event rate raises up to 20,000, stays at 
20,000 a while and drops to about 15000-17000. The same pattern

Processing time is just about 400 ms.

Any ideas/suggestions?

Thanks,
Patcharee

Re: spark streaming input rate strange

[Akhil Das <ak...@sigmoidanalytics.com>]

How are you verifying the data dropping? Can you send 10k, 20k events and
write the same to an output location from spark streaming and verify it? If
you are finding a data mismatch then its a problem with your
MulticastSocket implementation.

Thanks
Best Regards

On Fri, Jan 22, 2016 at 5:44 PM, patcharee <Pa...@uni.no>
wrote:

> Hi,
>
> I have a streaming application with
> - 1 sec interval
> - accept data from a simulation through MulticastSocket
>
> The simulation sent out data using multiple clients/threads every 1 sec
> interval. The input rate accepted by the streaming looks strange.
> - When clients = 10,000 the event rate raises up to 10,000, stays at
> 10,000 a while and drops to about 7000-8000.
> - When clients = 20,000 the event rate raises up to 20,000, stays at
> 20,000 a while and drops to about 15000-17000. The same pattern
>
> Processing time is just about 400 ms.
>
> Any ideas/suggestions?
>
> Thanks,
> Patcharee
>