You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by tkg_cangkul <yu...@gmail.com> on 2017/10/10 05:25:36 UTC
metron dashboard timeout when loads many data
Hi,
anyone have experienced with query heavy data on metron dashboard?
i have 30Gb data. But when i try to load it all with metron dashboard in
kibana, i have an error msg like below:
*Request Timeout after 30000ms
*and then after that i've got this error msg too :
*Caused by: EsRejectedExecutionException[rejected execution of
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2
on EsThreadPoolExecutor[search, queue capacity = 100,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running,
pool size = 20, active threads = 20, queued tasks = 100, completed tasks
= 192656]]]
*For your information,
my ES heap is 10Gb
with 3 master and 4 datanodes.
Pls advice,
Best Regards,
Re: metron dashboard timeout when loads many data
Posted by tkg_cangkul <yu...@gmail.com>.
Hi James,
I've try your suggestion about this but it doesn't works.
i've got msg master_not_discovered_exception oftenly.
On 17/10/17 02:48, James Sirota wrote:
> that is correct. please let us know if that works for you
>
>
> 12.10.2017, 10:14, "Youzha" <yu...@gmail.com>:
>> so i must restart the datanodes one by one. and set only one master
>> node right?
>> Ok i got it. thank you so much James for your explanation. i will try
>> it soon.
>>
>> On Thu, 12 Oct 2017 at 21.55 James Sirota <jsirota@apache.org
>> <ma...@apache.org>> wrote:
>>
>> You have to restart the ES cluster in a rolling fashion. Meaning
>> restart one data node, then the other, then the other, etc. If
>> you restart them all at once, this will happen
>>
>>
>> 11.10.2017, 19:37, "tkg_cangkul" <yuza.rasfar@gmail.com
>> <ma...@gmail.com>>:
>>> First time i install metron on cluster, i just only set one
>>> master node. But there are many problems when recovery starts.
>>> after i restart the ES cluster, sometimes there are some nodes
>>> that disconnected randomly from cluster when recovery proccess.
>>> When i set 3 master nodes, the disconnected node problem before
>>> was solved.
>>>
>>> On 12/10/17 08:13, James Sirota wrote:
>>>> Your problem is that all your master nodes are also your data
>>>> nodes. For such a small cluster you have to make one master
>>>> node and the rest of the nodes would be data nodes. Ideally you
>>>> don't want to colocate master nodes and data nodes. If you fix
>>>> that I think it should work
>>>>
>>>>
>>>> 10.10.2017, 19:50, "tkg_cangkul" <yu...@gmail.com>
>>>> <ma...@gmail.com>:
>>>>> Hi James,
>>>>>
>>>>> Thanks for your reply,
>>>>> This is the output of the API's :
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> For GET /_cluster/allocation/ API, there is an error like below :
>>>>>
>>>>>
>>>>>
>>>>> For your information, I'm using Elasticsearch 2.3.3
>>>>>
>>>>> On 10/10/17 23:49, James Sirota wrote:
>>>>>> I suspect your Elasticsearch may be in a bad state. If you
>>>>>> are using Chrome, can you download the sense plugin and then
>>>>>> run the following commands:
>>>>>> GET /_cluster/health?pretty
>>>>>> GET _cat/pending_tasks?v
>>>>>> GET /_cat/nodes?v
>>>>>> GET /_cluster/allocation/
>>>>>> And paste the output here?
>>>>>> Thanks,
>>>>>> James
>>>>>>
>>>>>>
>>>>>> 09.10.2017, 22:25, "tkg_cangkul" <yu...@gmail.com>
>>>>>> <ma...@gmail.com>:
>>>>>>> Hi,
>>>>>>>
>>>>>>> anyone have experienced with query heavy data on metron
>>>>>>> dashboard?
>>>>>>> i have 30Gb data. But when i try to load it all with metron
>>>>>>> dashboard in kibana, i have an error msg like below:
>>>>>>>
>>>>>>> *Request Timeout after 30000ms
>>>>>>>
>>>>>>> *and then after that i've got this error msg too :
>>>>>>>
>>>>>>> *Caused by: EsRejectedExecutionException[rejected execution
>>>>>>> of
>>>>>>> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2
>>>>>>> on EsThreadPoolExecutor[search, queue capacity = 100,
>>>>>>> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running,
>>>>>>> pool size = 20, active threads = 20, queued tasks = 100,
>>>>>>> completed tasks = 192656]]]
>>>>>>>
>>>>>>> *For your information,
>>>>>>> my ES heap is 10Gb
>>>>>>> with 3 master and 4 datanodes.
>>>>>>>
>>>>>>> Pls advice,
>>>>>>>
>>>>>>> Best Regards,
>>>>>>>
>>>>>>
>>>>>>
>>>>>> -------------------
>>>>>> Thank you,
>>>>>> James Sirota
>>>>>> PMC- Apache Metron
>>>>>> jsirota AT apache DOT org
>>>>>>
>>>>>
>>>>
>>>>
>>>> -------------------
>>>> Thank you,
>>>> James Sirota
>>>> PMC- Apache Metron
>>>> jsirota AT apache DOT org
>>>>
>>>
>>
>>
>> -------------------
>> Thank you,
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>
>
> -------------------
> Thank you,
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
Re: metron dashboard timeout when loads many data
Posted by Youzha <yu...@gmail.com>.
so i must restart the datanodes one by one. and set only one master node
right?
Ok i got it. thank you so much James for your explanation. i will try it
soon.
On Thu, 12 Oct 2017 at 21.55 James Sirota <js...@apache.org> wrote:
> You have to restart the ES cluster in a rolling fashion. Meaning restart
> one data node, then the other, then the other, etc. If you restart them all
> at once, this will happen
>
>
> 11.10.2017, 19:37, "tkg_cangkul" <yu...@gmail.com>:
>
> First time i install metron on cluster, i just only set one master node.
> But there are many problems when recovery starts. after i restart the ES
> cluster, sometimes there are some nodes that disconnected randomly from
> cluster when recovery proccess. When i set 3 master nodes, the disconnected
> node problem before was solved.
>
> On 12/10/17 08:13, James Sirota wrote:
>
> Your problem is that all your master nodes are also your data nodes. For
> such a small cluster you have to make one master node and the rest of the
> nodes would be data nodes. Ideally you don't want to colocate master nodes
> and data nodes. If you fix that I think it should work
>
>
> 10.10.2017, 19:50, "tkg_cangkul" <yu...@gmail.com>
> <yu...@gmail.com>:
>
> Hi James,
>
> Thanks for your reply,
> This is the output of the API's :
>
>
>
>
>
> For GET /_cluster/allocation/ API, there is an error like below :
>
>
>
> For your information, I'm using Elasticsearch 2.3.3
>
> On 10/10/17 23:49, James Sirota wrote:
>
> I suspect your Elasticsearch may be in a bad state. If you are using
> Chrome, can you download the sense plugin and then run the following
> commands:
>
> GET /_cluster/health?pretty
>
> GET _cat/pending_tasks?v
>
> GET /_cat/nodes?v
>
> GET /_cluster/allocation/
>
> And paste the output here?
>
> Thanks,
> James
>
>
> 09.10.2017, 22:25, "tkg_cangkul" <yu...@gmail.com>
> <yu...@gmail.com>:
>
> Hi,
>
> anyone have experienced with query heavy data on metron dashboard?
> i have 30Gb data. But when i try to load it all with metron dashboard in
> kibana, i have an error msg like below:
>
>
>
> *Request Timeout after 30000ms *and then after that i've got this error
> msg too :
>
>
>
> *Caused by: EsRejectedExecutionException[rejected execution of
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2
> on EsThreadPoolExecutor[search, queue capacity = 100,
> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running,
> pool size = 20, active threads = 20, queued tasks = 100, completed tasks =
> 192656]]] *For your information,
> my ES heap is 10Gb
> with 3 master and 4 datanodes.
>
> Pls advice,
>
> Best Regards,
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
Re: metron dashboard timeout when loads many data
Posted by tkg_cangkul <yu...@gmail.com>.
First time i install metron on cluster, i just only set one master node.
But there are many problems when recovery starts. after i restart the ES
cluster, sometimes there are some nodes that disconnected randomly from
cluster when recovery proccess. When i set 3 master nodes, the
disconnected node problem before was solved.
On 12/10/17 08:13, James Sirota wrote:
> Your problem is that all your master nodes are also your data nodes.
> For such a small cluster you have to make one master node and the rest
> of the nodes would be data nodes. Ideally you don't want to colocate
> master nodes and data nodes. If you fix that I think it should work
>
>
> 10.10.2017, 19:50, "tkg_cangkul" <yu...@gmail.com>:
>> Hi James,
>>
>> Thanks for your reply,
>> This is the output of the API's :
>>
>>
>>
>>
>>
>> For GET /_cluster/allocation/ API, there is an error like below :
>>
>>
>>
>> For your information, I'm using Elasticsearch 2.3.3
>>
>> On 10/10/17 23:49, James Sirota wrote:
>>> I suspect your Elasticsearch may be in a bad state. If you are using
>>> Chrome, can you download the sense plugin and then run the following
>>> commands:
>>> GET /_cluster/health?pretty
>>> GET _cat/pending_tasks?v
>>> GET /_cat/nodes?v
>>> GET /_cluster/allocation/
>>> And paste the output here?
>>> Thanks,
>>> James
>>>
>>>
>>> 09.10.2017, 22:25, "tkg_cangkul" <yu...@gmail.com>
>>> <ma...@gmail.com>:
>>>> Hi,
>>>>
>>>> anyone have experienced with query heavy data on metron dashboard?
>>>> i have 30Gb data. But when i try to load it all with metron
>>>> dashboard in kibana, i have an error msg like below:
>>>>
>>>> *Request Timeout after 30000ms
>>>>
>>>> *and then after that i've got this error msg too :
>>>>
>>>> *Caused by: EsRejectedExecutionException[rejected execution of
>>>> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2
>>>> on EsThreadPoolExecutor[search, queue capacity = 100,
>>>> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running,
>>>> pool size = 20, active threads = 20, queued tasks = 100, completed
>>>> tasks = 192656]]]
>>>>
>>>> *For your information,
>>>> my ES heap is 10Gb
>>>> with 3 master and 4 datanodes.
>>>>
>>>> Pls advice,
>>>>
>>>> Best Regards,
>>>>
>>>
>>>
>>> -------------------
>>> Thank you,
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
>>>
>>
>
>
> -------------------
> Thank you,
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
Re: metron dashboard timeout when loads many data
Posted by tkg_cangkul <yu...@gmail.com>.
Hi James,
Thanks for your reply,
This is the output of the API's :
For GET /_cluster/allocation/ API, there is an error like below :
For your information, I'm using Elasticsearch 2.3.3
On 10/10/17 23:49, James Sirota wrote:
> I suspect your Elasticsearch may be in a bad state. If you are using
> Chrome, can you download the sense plugin and then run the following
> commands:
> GET /_cluster/health?pretty
> GET _cat/pending_tasks?v
> GET /_cat/nodes?v
> GET /_cluster/allocation/
> And paste the output here?
> Thanks,
> James
>
>
> 09.10.2017, 22:25, "tkg_cangkul" <yu...@gmail.com>:
>> Hi,
>>
>> anyone have experienced with query heavy data on metron dashboard?
>> i have 30Gb data. But when i try to load it all with metron dashboard
>> in kibana, i have an error msg like below:
>>
>> *Request Timeout after 30000ms
>>
>> *and then after that i've got this error msg too :
>>
>> *Caused by: EsRejectedExecutionException[rejected execution of
>> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@eba98f2
>> on EsThreadPoolExecutor[search, queue capacity = 100,
>> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@52fd7ae9[Running,
>> pool size = 20, active threads = 20, queued tasks = 100, completed
>> tasks = 192656]]]
>>
>> *For your information,
>> my ES heap is 10Gb
>> with 3 master and 4 datanodes.
>>
>> Pls advice,
>>
>> Best Regards,
>>
>
>
> -------------------
> Thank you,
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>