You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by satyavrat <bi...@gmail.com> on 2015/07/27 10:55:15 UTC
Trident DRPC not distributing task to all workers
Hi,
I am using DRPC for my log processing.
We have a very heavy task and trying to do it with drpc.
Input to drpc is a file name , output is usually 1GB string data.
Problem : DRPC is not assigning task to all workers.
Currently running 5 nodes.Other 4 nodes using given configuration server as
drpc.
Some workers will get job immediately but some of them not getting assigned
in 20-30 mins.
Kindly help.
Configuration is :
storm.zookeeper.session.timeout: 250000
storm.zookeeper.connection.timeout: 250000
nimbus.task.timeout.secs: 300
nimbus.monitor.freq.secs: 10
storm.local.dir: "/mnt/storm"
supervisor.worker.timeout.secs: 250
worker.heartbeat.frequency.secs: 5
supervisor.monitor.frequency.secs: 10
supervisor.heartbeat.frequency.secs: 5
drpc.request.timeout.secs: 3500
drpc.worker.threads: 640
drpc.max_buffer_size: 10485760
drpc.queue.size: 1280
drpc.invocations.threads: 640
drpc.childopts: "-Xmx7680m"
drpc.servers:
- "localhost"
storm.zookeeper.servers:
- "localhost"
nimbus.host: "localhost"
storm.local.dir: "/mnt/storm"
worker.childopts: "-Xmx25000m"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703..more workers
Thanks
Satyavrat
Re: Trident DRPC not distributing task to all workers
Posted by satyavrat <bi...@gmail.com>.
Hi all,
Please help.
On Jul 27, 2015 2:25 PM, "satyavrat" <bi...@gmail.com> wrote:
> Hi,
>
> I am using DRPC for my log processing.
> We have a very heavy task and trying to do it with drpc.
> Input to drpc is a file name , output is usually 1GB string data.
>
> Problem : DRPC is not assigning task to all workers.
> Currently running 5 nodes.Other 4 nodes using given configuration server
> as drpc.
> Some workers will get job immediately but some of them not getting
> assigned in 20-30 mins.
>
> Kindly help.
>
> Configuration is :
> storm.zookeeper.session.timeout: 250000
> storm.zookeeper.connection.timeout: 250000
> nimbus.task.timeout.secs: 300
> nimbus.monitor.freq.secs: 10
> storm.local.dir: "/mnt/storm"
> supervisor.worker.timeout.secs: 250
> worker.heartbeat.frequency.secs: 5
> supervisor.monitor.frequency.secs: 10
> supervisor.heartbeat.frequency.secs: 5
>
> drpc.request.timeout.secs: 3500
> drpc.worker.threads: 640
> drpc.max_buffer_size: 10485760
> drpc.queue.size: 1280
> drpc.invocations.threads: 640
> drpc.childopts: "-Xmx7680m"
>
> drpc.servers:
> - "localhost"
>
> storm.zookeeper.servers:
> - "localhost"
>
> nimbus.host: "localhost"
> storm.local.dir: "/mnt/storm"
> worker.childopts: "-Xmx25000m"
> supervisor.slots.ports:
> - 6700
> - 6701
> - 6702
> - 6703..more workers
>
>
> Thanks
> Satyavrat
>
Fwd: Trident DRPC not distributing task to all workers
Posted by satyavrat <bi...@gmail.com>.
Hi,
I am using storm 0.8.2 , trident DRPC for my log processing.
We have a very heavy task and trying to do it with drpc.
Input to drpc is a file name , output is usually 1GB string data.
Problem : DRPC is not assigning task to all workers.
Currently running 5 nodes.Other 4 nodes using given configuration server as
drpc.
Some workers will get job immediately but some of them not getting assigned
in 20-30 mins.
Kindly help.
Configuration is :
storm.zookeeper.session.timeout: 250000
storm.zookeeper.connection.timeout: 250000
nimbus.task.timeout.secs: 300
nimbus.monitor.freq.secs: 10
storm.local.dir: "/mnt/storm"
supervisor.worker.timeout.secs: 250
worker.heartbeat.frequency.secs: 5
supervisor.monitor.frequency.secs: 10
supervisor.heartbeat.frequency.secs: 5
drpc.request.timeout.secs: 3500
drpc.worker.threads: 640
drpc.max_buffer_size: 10485760
drpc.queue.size: 1280
drpc.invocations.threads: 640
drpc.childopts: "-Xmx7680m"
drpc.servers:
- "localhost"
storm.zookeeper.servers:
- "localhost"
nimbus.host: "localhost"
storm.local.dir: "/mnt/storm"
worker.childopts: "-Xmx25000m"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703..more workers
Thanks
Satyavrat