Re: Σχετ: DRPC and Trident Throughput

[Jonas Krauß <jk...@gmail.com>]

Hello,

thanks for the advice, I think the behavior you describe explains why Trident becomes slow when using multiple workers. This is a pity because as the documentation states you do not have direct influence on how the batch size is set by Trident. However, to me it still remains unclear why it becomes THAT slow (1/10 of the speed with a single worker setup). I should probably add the all servers of the cluster we run are located in the same place and have a ping of 1ms or less(!) between them.

I was thinking of using Kafka/Redis or a similar pub/sub system, however, to me it seemed intriguing if I could just use the internal DRPC server without implementing another new technology. I have not given up completely yet: experimenting more I found that when I increase the parallelism IN TOPOLOGIES DOING THE DRPC EXECUTES the throughput increases up to a level where we at least start to see the available capacity in the bolts being exhausted (that is in the single worker setup).

I still would appreciate any pointers to a good documentation on how the DRPC <--> Trident relationship works in the background. I will try to find the right place in the source code, a pointer to the correct file would be helpful as well. Also I would like to know why multiple workers with increased parallelism actually build up congestion rather than increasing throughput, this seems to be a separate issue.

Thanks

Jonas


On Jul 28, 2017, at 8:18 PM, kalliopi kalantzaki <ts...@yahoo.gr> wrote:

> Hi,
> 
> Check whether the whole topology is slow due to processing burden... The problem with DRPC is that if one thread/call is slow (e.g tuple fails or passes through a slow bolt etc) then the following calls will get slow as well--I think because the system waits until the current call finishes... I am not sure about the last one, check it if you want. 
> 
> The most common solution i have seen in such cases, is to put in front of storm a kafka system where the incoming call writes to a topic, storm then reads and processes the incoming call and then writes back the result to another topic via a kafkabolt. From the output topic then you can implement a producer to take the results....
> This solution definitely works but again maybe this does not fit your use cases....
> 
> Hope this helps.
>  
> 
> 
> 
> Στις 5:17 μ.μ. Παρασκευή, 28 Ιουλίου 2017, ο/η Jonas Krauß <jk...@gmail.com> έγραψε:
> 
> 
> Hi all,
> 
> I am trying to understand how storm's DRPC server and Trident work together. A particular Trident topology I am developing seems to be limited in its throughput as batches/tuples are not pushed into it fast enough from the DRPC server. We are using storm 1.1.0.
> 
> Unfortunately I have not found a good source of information on how the DRPC server and Trident function together, though I have looked at the various Trident examples and docs at https://storm.apache.org/releases/1.1.0/index.html as well as Stackoverflow and the old storm Google Group.
> 
> What I have tried so far:
> - lowering the trident batch emit interval millis from 500 to 200, 100 and 25, does not seem to have an effect
> - using multiple workers in the Trident topology along with different kinds of partitioning the stream - seems to hurt performance (I suspect network latency between our storm servers to be the reason for that)
> - using a single or multiple spouts, seems to have no effect
> 
> Our drpc is configured as follows:
> 
> drpc.authorizer.acl.filename    "drpc-auth-acl.yaml"
> drpc.authorizer.acl.strict    false
> drpc.childopts    "-Xmx2048m"
> drpc.http.creds.plugin    "org.apache.storm.security.auth.DefaultHttpCredentialsPlugin"
> drpc.http.port    3774
> drpc.https.keystore.password    ""
> drpc.https.keystore.type    "JKS"
> drpc.https.port    -1
> drpc.invocations.port    3773
> drpc.invocations.threads    64
> drpc.max_buffer_size    10485760
> drpc.port    3772
> drpc.queue.size    128
> drpc.request.timeout.secs    600
> drpc.worker.threads    64
> 
> I appreciate any pointers to a good documentation or concrete suggestions. Sorry if this question has been asked before.
> 
> Thanks
> 
> Jonas
> 
>