You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by tkg_cangkul <yu...@gmail.com> on 2017/03/29 04:55:52 UTC
tuning indexing metron
hi Zeolla,
thanks before for your reply.
sorry if i changed the email subject. i think the problem of kafka topic
partition has been solved. but the indexing still slow for me.
this is my cluster environtment for ES:
i have 5 servers ( 1 master, 4 datanode )
every server there is 64Gb memory on each machine.
i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but the
indexing still slow.
any suggest to tuning the indexing?
On 28/03/17 08:43, Zeolla@GMail.com wrote:
> Can you clarify what you mean by recreate your kafka topics? Usually
> what I do to add partitions to a kafka topic in Metron is something like:
>
> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
> --zookeeper $zk --alter --topic bro --partitions 4
>
> Once you run this, your bro topic (or whatever you specify next to
> topic) will now have multiple partitions. It is effectively
> transparent to the rest of Metron.
>
> That said, are you running kafka and ES on the same boxes? You say you
> want 4 partitions because you have 4 ES boxes, but it should really
> map to the # of boxes/disks you have running kafka brokers. Would you
> mind providing a basic layout of your environment (what servers run
> where, how many servers, etc.)? Thanks,
>
> Jon
>
> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul <yuza.rasfar@gmail.com
> <ma...@gmail.com>> wrote:
>
> anyone can help me for solving this problem?
>
>
> On 24/03/17 18:24, tkg_cangkul wrote:
>> hi,
>>
>> i wanna try to tuning my ES when indexing data from all the
>> sensors (bro, yaf, snort).
>> i've read this article :
>>
>> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
>>
>> on point 3. Assuming that ES will get behind from time to time,
>> you may want to increase the indexing kafka topic size. You
>> should also consider increasing the partitioning kafka topics in
>> general to distribute the load better and increase parallelism.
>>
>> then i try to recreate my kafka topic to 4 partition because i
>> have 4 datanode on my elastic search. then, i check the broker
>> node in zookeeper for all the topics that i create and i've seen
>> there are *[1,2,3]* partition inside the
>> */brokers/topics/(all_my_topics_name)/partitions*
>> But when i start the storm topology, i've seen this error message
>> on my storm logs if there is no node for
>> */brokers/topics/(all_my_topics_name)/partitions/0/state
>>
>> *Why it still read the *0 *partitions.? is the metron just set
>> for assigned 1 partition only on kafka topic?
>> then i try to create manually the
>> */brokers/topics/(all_my_topics_name)/partitions/0/state *but
>> i've found another error msg like this:
>>
>>
>>
>> any suggestion about this?
>>
>>
>>
>
> --
>
> Jon
>
Re: tuning indexing metron
Posted by Nick Allen <ni...@nickallen.org>.
How are you measuring performance? What performance are you seeing? What
performance do you expect to see?
It is really hard to tune effectively if you have other apps running on the
same host. You need to isolate the environment, at least during this
exercise.
On Wed, Mar 29, 2017 at 12:02 PM, tkg_cangkul <yu...@gmail.com> wrote:
> Well,
>
> actually there are not only metron that running on my cluster. there are
> some other apps running on it. i'm afraid it will be crash if i set it to
> 50% of my RAM. Are ES heap ideally set to 50% RAM memory to get the maximum
> performance?
>
>
>
> On 29/03/17 17:18, Zeolla@GMail.com wrote:
>
> Right off the bat I would give 31GB heap to each ES node. Normally you
> want that number close to 32GB but not exceeding 50% of your total server
> RAM so it can be used for disk caching. Let me know if that helps, I'd be
> happy to help further.
>
> Jon
>
> On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yu...@gmail.com> wrote:
>
>> hi Zeolla,
>>
>> thanks before for your reply.
>> sorry if i changed the email subject. i think the problem of kafka topic
>> partition has been solved. but the indexing still slow for me.
>>
>> this is my cluster environtment for ES:
>> i have 5 servers ( 1 master, 4 datanode )
>> every server there is 64Gb memory on each machine.
>>
>> i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but the
>> indexing still slow.
>>
>> any suggest to tuning the indexing?
>>
>>
>> On 28/03/17 08:43, Zeolla@GMail.com wrote:
>>
>> Can you clarify what you mean by recreate your kafka topics? Usually
>> what I do to add partitions to a kafka topic in Metron is something like:
>>
>> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
>> --zookeeper $zk --alter --topic bro --partitions 4
>>
>> Once you run this, your bro topic (or whatever you specify next to topic)
>> will now have multiple partitions. It is effectively transparent to the
>> rest of Metron.
>>
>> That said, are you running kafka and ES on the same boxes? You say you
>> want 4 partitions because you have 4 ES boxes, but it should really map to
>> the # of boxes/disks you have running kafka brokers. Would you mind
>> providing a basic layout of your environment (what servers run where, how
>> many servers, etc.)? Thanks,
>>
>> Jon
>>
>> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul <yu...@gmail.com>
>> wrote:
>>
>> anyone can help me for solving this problem?
>>
>>
>> On 24/03/17 18:24, tkg_cangkul wrote:
>>
>> hi,
>>
>> i wanna try to tuning my ES when indexing data from all the sensors (bro,
>> yaf, snort).
>> i've read this article :
>>
>> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
>>
>> on point 3. Assuming that ES will get behind from time to time, you may
>> want to increase the indexing kafka topic size. You should also consider
>> increasing the partitioning kafka topics in general to distribute the load
>> better and increase parallelism.
>>
>> then i try to recreate my kafka topic to 4 partition because i have 4
>> datanode on my elastic search. then, i check the broker node in zookeeper
>> for all the topics that i create and i've seen there are *[1,2,3]*
>> partition inside the */brokers/topics/(all_my_topics_name)/partitions*
>> But when i start the storm topology, i've seen this error message on my
>> storm logs if there is no node for
>>
>> */brokers/topics/(all_my_topics_name)/partitions/0/state *Why it still
>> read the *0 *partitions.? is the metron just set for assigned 1
>> partition only on kafka topic?
>> then i try to create manually the */brokers/topics/(all_my_topics_name)/partitions/0/state
>> *but i've found another error msg like this:
>>
>>
>>
>> any suggestion about this?
>>
>>
>>
>>
>> --
>>
>> Jon
>>
>>
>> --
>
> Jon
>
>
>
Re: tuning indexing metron
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Sounds good, let us know how it goes. There are many tuning options with
ES and in Metron, a lot of which are described in the guide I put together,
but some are definitely missing. That said, the guide could use an update
and if you can think of anything that would help make it more clear I'd be
happy to clean it up. I also have on my to do list to move it into the
GitHub code, and thus the generated site-book(s).
Jon
On Wed, Mar 29, 2017, 1:06 PM tkg_cangkul <yu...@gmail.com> wrote:
> hmm ok.
>
> so it means i better run this metron on separate cluster with my other
> apps right?
>
> ok thank you so much for all of your help guys. :)
> i'll try it soon after i migrate my other apps.
>
>
>
> On 29/03/17 23:52, Simon Elliston Ball wrote:
>
> In general for any ES use case you should be running only ES on the ES
> nodes, and yes, you should use min(31, RAM / 2) as heap size. It’s also
> worth considering shard counts for indices and disk layout in the ES nodes.
>
> As Nick points out, you’re also going to find it hard to tune ES for large
> numbers of applications running different indexing, so it’s generally a
> good idea to build different ES environments for your multiple use cases.
>
> Simon
>
> On 29 Mar 2017, at 17:02, tkg_cangkul <yu...@gmail.com> wrote:
>
> Well,
>
> actually there are not only metron that running on my cluster. there are
> some other apps running on it. i'm afraid it will be crash if i set it to
> 50% of my RAM. Are ES heap ideally set to 50% RAM memory to get the maximum
> performance?
>
>
> On 29/03/17 17:18, Zeolla@GMail.com wrote:
>
> Right off the bat I would give 31GB heap to each ES node. Normally you
> want that number close to 32GB but not exceeding 50% of your total server
> RAM so it can be used for disk caching. Let me know if that helps, I'd be
> happy to help further.
>
> Jon
>
> On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yu...@gmail.com> wrote:
>
> hi Zeolla,
>
> thanks before for your reply.
> sorry if i changed the email subject. i think the problem of kafka topic
> partition has been solved. but the indexing still slow for me.
>
> this is my cluster environtment for ES:
> i have 5 servers ( 1 master, 4 datanode )
> every server there is 64Gb memory on each machine.
>
> i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but the
> indexing still slow.
>
> any suggest to tuning the indexing?
>
>
> On 28/03/17 08:43, Zeolla@GMail.com wrote:
>
> Can you clarify what you mean by recreate your kafka topics? Usually what
> I do to add partitions to a kafka topic in Metron is something like:
>
> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
> --zookeeper $zk --alter --topic bro --partitions 4
>
> Once you run this, your bro topic (or whatever you specify next to topic)
> will now have multiple partitions. It is effectively transparent to the
> rest of Metron.
>
> That said, are you running kafka and ES on the same boxes? You say you
> want 4 partitions because you have 4 ES boxes, but it should really map to
> the # of boxes/disks you have running kafka brokers. Would you mind
> providing a basic layout of your environment (what servers run where, how
> many servers, etc.)? Thanks,
>
> Jon
>
> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul <yu...@gmail.com>
> wrote:
>
> anyone can help me for solving this problem?
>
>
> On 24/03/17 18:24, tkg_cangkul wrote:
>
> hi,
>
> i wanna try to tuning my ES when indexing data from all the sensors (bro,
> yaf, snort).
> i've read this article :
>
> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
>
> on point 3. Assuming that ES will get behind from time to time, you may
> want to increase the indexing kafka topic size. You should also consider
> increasing the partitioning kafka topics in general to distribute the load
> better and increase parallelism.
>
> then i try to recreate my kafka topic to 4 partition because i have 4
> datanode on my elastic search. then, i check the broker node in zookeeper
> for all the topics that i create and i've seen there are *[1,2,3]*
> partition inside the */brokers/topics/(all_my_topics_name)/partitions*
> But when i start the storm topology, i've seen this error message on my
> storm logs if there is no node for
>
> */brokers/topics/(all_my_topics_name)/partitions/0/state *Why it still
> read the *0 *partitions.? is the metron just set for assigned 1 partition
> only on kafka topic?
> then i try to create manually the */brokers/topics/(all_my_topics_name)/partitions/0/state
> *but i've found another error msg like this:
>
>
>
> any suggestion about this?
>
>
>
>
> --
>
> Jon
>
>
> --
>
> Jon
>
>
>
>
> --
Jon
Re: tuning indexing metron
Posted by tkg_cangkul <yu...@gmail.com>.
hmm ok.
so it means i better run this metron on separate cluster with my other
apps right?
ok thank you so much for all of your help guys. :)
i'll try it soon after i migrate my other apps.
On 29/03/17 23:52, Simon Elliston Ball wrote:
> In general for any ES use case you should be running only ES on the ES
> nodes, and yes, you should use min(31, RAM / 2) as heap size. It\u2019s
> also worth considering shard counts for indices and disk layout in the
> ES nodes.
>
> As Nick points out, you\u2019re also going to find it hard to tune ES for
> large numbers of applications running different indexing, so it\u2019s
> generally a good idea to build different ES environments for your
> multiple use cases.
>
> Simon
>
>> On 29 Mar 2017, at 17:02, tkg_cangkul <yuza.rasfar@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Well,
>>
>> actually there are not only metron that running on my cluster. there
>> are some other apps running on it. i'm afraid it will be crash if i
>> set it to 50% of my RAM. Are ES heap ideally set to 50% RAM memory to
>> get the maximum performance?
>>
>>
>> On 29/03/17 17:18, Zeolla@GMail.com wrote:
>>>
>>> Right off the bat I would give 31GB heap to each ES node. Normally
>>> you want that number close to 32GB but not exceeding 50% of your
>>> total server RAM so it can be used for disk caching. Let me know if
>>> that helps, I'd be happy to help further.
>>>
>>> Jon
>>>
>>>
>>> On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yuza.rasfar@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> hi Zeolla,
>>>
>>> thanks before for your reply.
>>> sorry if i changed the email subject. i think the problem of
>>> kafka topic partition has been solved. but the indexing still
>>> slow for me.
>>>
>>> this is my cluster environtment for ES:
>>> i have 5 servers ( 1 master, 4 datanode )
>>> every server there is 64Gb memory on each machine.
>>>
>>> i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic
>>> but the indexing still slow.
>>>
>>> any suggest to tuning the indexing?
>>>
>>>
>>> On 28/03/17 08:43, Zeolla@GMail.com <ma...@GMail.com> wrote:
>>>> Can you clarify what you mean by recreate your kafka topics?
>>>> Usually what I do to add partitions to a kafka topic in Metron
>>>> is something like:
>>>>
>>>> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
>>>> --zookeeper $zk --alter --topic bro --partitions 4
>>>>
>>>> Once you run this, your bro topic (or whatever you specify next
>>>> to topic) will now have multiple partitions. It is effectively
>>>> transparent to the rest of Metron.
>>>>
>>>> That said, are you running kafka and ES on the same boxes? You
>>>> say you want 4 partitions because you have 4 ES boxes, but it
>>>> should really map to the # of boxes/disks you have running
>>>> kafka brokers. Would you mind providing a basic layout of your
>>>> environment (what servers run where, how many servers, etc.)?
>>>> Thanks,
>>>>
>>>> Jon
>>>>
>>>> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul
>>>> <yuza.rasfar@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> anyone can help me for solving this problem?
>>>>
>>>>
>>>> On 24/03/17 18:24, tkg_cangkul wrote:
>>>>> hi,
>>>>>
>>>>> i wanna try to tuning my ES when indexing data from all
>>>>> the sensors (bro, yaf, snort).
>>>>> i've read this article :
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
>>>>>
>>>>> on point 3. Assuming that ES will get behind from time to
>>>>> time, you may want to increase the indexing kafka topic
>>>>> size. You should also consider increasing the partitioning
>>>>> kafka topics in general to distribute the load better and
>>>>> increase parallelism.
>>>>>
>>>>> then i try to recreate my kafka topic to 4 partition
>>>>> because i have 4 datanode on my elastic search. then, i
>>>>> check the broker node in zookeeper for all the topics that
>>>>> i create and i've seen there are *[1,2,3]* partition
>>>>> inside the */brokers/topics/(all_my_topics_name)/partitions*
>>>>> But when i start the storm topology, i've seen this error
>>>>> message on my storm logs if there is no node for
>>>>> */brokers/topics/(all_my_topics_name)/partitions/0/state
>>>>>
>>>>> *Why it still read the *0 *partitions.? is the metron just
>>>>> set for assigned 1 partition only on kafka topic?
>>>>> then i try to create manually the
>>>>> */brokers/topics/(all_my_topics_name)/partitions/0/state
>>>>> *but i've found another error msg like this:
>>>>>
>>>>>
>>>>>
>>>>> any suggestion about this?
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Jon
>>>>
>>>
>>> --
>>>
>>> Jon
>>>
>>
>
Re: tuning indexing metron
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
In general for any ES use case you should be running only ES on the ES nodes, and yes, you should use min(31, RAM / 2) as heap size. It’s also worth considering shard counts for indices and disk layout in the ES nodes.
As Nick points out, you’re also going to find it hard to tune ES for large numbers of applications running different indexing, so it’s generally a good idea to build different ES environments for your multiple use cases.
Simon
> On 29 Mar 2017, at 17:02, tkg_cangkul <yu...@gmail.com> wrote:
>
> Well,
>
> actually there are not only metron that running on my cluster. there are some other apps running on it. i'm afraid it will be crash if i set it to 50% of my RAM. Are ES heap ideally set to 50% RAM memory to get the maximum performance?
>
>
> On 29/03/17 17:18, Zeolla@GMail.com <ma...@GMail.com> wrote:
>> Right off the bat I would give 31GB heap to each ES node. Normally you want that number close to 32GB but not exceeding 50% of your total server RAM so it can be used for disk caching. Let me know if that helps, I'd be happy to help further.
>>
>> Jon
>>
>>
>> On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yuza.rasfar@gmail.com <ma...@gmail.com>> wrote:
>> hi Zeolla,
>>
>> thanks before for your reply.
>> sorry if i changed the email subject. i think the problem of kafka topic partition has been solved. but the indexing still slow for me.
>>
>> this is my cluster environtment for ES:
>> i have 5 servers ( 1 master, 4 datanode )
>> every server there is 64Gb memory on each machine.
>>
>> i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but the indexing still slow.
>>
>> any suggest to tuning the indexing?
>>
>>
>> On 28/03/17 08:43, Zeolla@GMail.com <ma...@GMail.com> wrote:
>>> Can you clarify what you mean by recreate your kafka topics? Usually what I do to add partitions to a kafka topic in Metron is something like:
>>>
>>> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --zookeeper $zk --alter --topic bro --partitions 4
>>>
>>> Once you run this, your bro topic (or whatever you specify next to topic) will now have multiple partitions. It is effectively transparent to the rest of Metron.
>>>
>>> That said, are you running kafka and ES on the same boxes? You say you want 4 partitions because you have 4 ES boxes, but it should really map to the # of boxes/disks you have running kafka brokers. Would you mind providing a basic layout of your environment (what servers run where, how many servers, etc.)? Thanks,
>>>
>>> Jon
>>>
>>> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul <yuza.rasfar@gmail.com <ma...@gmail.com>> wrote:
>>> anyone can help me for solving this problem?
>>>
>>>
>>> On 24/03/17 18:24, tkg_cangkul wrote:
>>>> hi,
>>>>
>>>> i wanna try to tuning my ES when indexing data from all the sensors (bro, yaf, snort).
>>>> i've read this article :
>>>>
>>>> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier <https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier>
>>>>
>>>> on point 3. Assuming that ES will get behind from time to time, you may want to increase the indexing kafka topic size. You should also consider increasing the partitioning kafka topics in general to distribute the load better and increase parallelism.
>>>>
>>>> then i try to recreate my kafka topic to 4 partition because i have 4 datanode on my elastic search. then, i check the broker node in zookeeper for all the topics that i create and i've seen there are [1,2,3] partition inside the /brokers/topics/(all_my_topics_name)/partitions
>>>> But when i start the storm topology, i've seen this error message on my storm logs if there is no node for /brokers/topics/(all_my_topics_name)/partitions/0/state
>>>>
>>>> Why it still read the 0 partitions.? is the metron just set for assigned 1 partition only on kafka topic?
>>>> then i try to create manually the /brokers/topics/(all_my_topics_name)/partitions/0/state but i've found another error msg like this:
>>>>
>>>>
>>>>
>>>> any suggestion about this?
>>>>
>>>>
>>>>
>>>
>>> --
>>> Jon
>>>
>>
>> --
>> Jon
>>
>
Re: tuning indexing metron
Posted by tkg_cangkul <yu...@gmail.com>.
Well,
actually there are not only metron that running on my cluster. there are
some other apps running on it. i'm afraid it will be crash if i set it
to 50% of my RAM. Are ES heap ideally set to 50% RAM memory to get the
maximum performance?
On 29/03/17 17:18, Zeolla@GMail.com wrote:
>
> Right off the bat I would give 31GB heap to each ES node. Normally
> you want that number close to 32GB but not exceeding 50% of your total
> server RAM so it can be used for disk caching. Let me know if that
> helps, I'd be happy to help further.
>
> Jon
>
>
> On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yuza.rasfar@gmail.com
> <ma...@gmail.com>> wrote:
>
> hi Zeolla,
>
> thanks before for your reply.
> sorry if i changed the email subject. i think the problem of kafka
> topic partition has been solved. but the indexing still slow for me.
>
> this is my cluster environtment for ES:
> i have 5 servers ( 1 master, 4 datanode )
> every server there is 64Gb memory on each machine.
>
> i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but
> the indexing still slow.
>
> any suggest to tuning the indexing?
>
>
> On 28/03/17 08:43, Zeolla@GMail.com <ma...@GMail.com> wrote:
>> Can you clarify what you mean by recreate your kafka topics?
>> Usually what I do to add partitions to a kafka topic in Metron is
>> something like:
>>
>> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
>> --zookeeper $zk --alter --topic bro --partitions 4
>>
>> Once you run this, your bro topic (or whatever you specify next
>> to topic) will now have multiple partitions. It is effectively
>> transparent to the rest of Metron.
>>
>> That said, are you running kafka and ES on the same boxes? You
>> say you want 4 partitions because you have 4 ES boxes, but it
>> should really map to the # of boxes/disks you have running kafka
>> brokers. Would you mind providing a basic layout of your
>> environment (what servers run where, how many servers, etc.)?
>> Thanks,
>>
>> Jon
>>
>> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul
>> <yuza.rasfar@gmail.com <ma...@gmail.com>> wrote:
>>
>> anyone can help me for solving this problem?
>>
>>
>> On 24/03/17 18:24, tkg_cangkul wrote:
>>> hi,
>>>
>>> i wanna try to tuning my ES when indexing data from all the
>>> sensors (bro, yaf, snort).
>>> i've read this article :
>>>
>>> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
>>>
>>> on point 3. Assuming that ES will get behind from time to
>>> time, you may want to increase the indexing kafka topic
>>> size. You should also consider increasing the partitioning
>>> kafka topics in general to distribute the load better and
>>> increase parallelism.
>>>
>>> then i try to recreate my kafka topic to 4 partition because
>>> i have 4 datanode on my elastic search. then, i check the
>>> broker node in zookeeper for all the topics that i create
>>> and i've seen there are *[1,2,3]* partition inside the
>>> */brokers/topics/(all_my_topics_name)/partitions*
>>> But when i start the storm topology, i've seen this error
>>> message on my storm logs if there is no node for
>>> */brokers/topics/(all_my_topics_name)/partitions/0/state
>>>
>>> *Why it still read the *0 *partitions.? is the metron just
>>> set for assigned 1 partition only on kafka topic?
>>> then i try to create manually the
>>> */brokers/topics/(all_my_topics_name)/partitions/0/state
>>> *but i've found another error msg like this:
>>>
>>>
>>>
>>> any suggestion about this?
>>>
>>>
>>>
>>
>> --
>>
>> Jon
>>
>
> --
>
> Jon
>
Re: tuning indexing metron
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Right off the bat I would give 31GB heap to each ES node. Normally you
want that number close to 32GB but not exceeding 50% of your total server
RAM so it can be used for disk caching. Let me know if that helps, I'd be
happy to help further.
Jon
On Wed, Mar 29, 2017, 12:56 AM tkg_cangkul <yu...@gmail.com> wrote:
> hi Zeolla,
>
> thanks before for your reply.
> sorry if i changed the email subject. i think the problem of kafka topic
> partition has been solved. but the indexing still slow for me.
>
> this is my cluster environtment for ES:
> i have 5 servers ( 1 master, 4 datanode )
> every server there is 64Gb memory on each machine.
>
> i have set 4Gb to my ES Heap Size. 4 Partition of kafka topic but the
> indexing still slow.
>
> any suggest to tuning the indexing?
>
>
> On 28/03/17 08:43, Zeolla@GMail.com wrote:
>
> Can you clarify what you mean by recreate your kafka topics? Usually what
> I do to add partitions to a kafka topic in Metron is something like:
>
> zk=server1:2181;/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh
> --zookeeper $zk --alter --topic bro --partitions 4
>
> Once you run this, your bro topic (or whatever you specify next to topic)
> will now have multiple partitions. It is effectively transparent to the
> rest of Metron.
>
> That said, are you running kafka and ES on the same boxes? You say you
> want 4 partitions because you have 4 ES boxes, but it should really map to
> the # of boxes/disks you have running kafka brokers. Would you mind
> providing a basic layout of your environment (what servers run where, how
> many servers, etc.)? Thanks,
>
> Jon
>
> On Fri, Mar 24, 2017 at 11:56 AM tkg_cangkul <yu...@gmail.com>
> wrote:
>
> anyone can help me for solving this problem?
>
>
> On 24/03/17 18:24, tkg_cangkul wrote:
>
> hi,
>
> i wanna try to tuning my ES when indexing data from all the sensors (bro,
> yaf, snort).
> i've read this article :
>
> https://cwiki.apache.org/confluence/display/METRON/Tuning+the+Search+tier
>
> on point 3. Assuming that ES will get behind from time to time, you may
> want to increase the indexing kafka topic size. You should also consider
> increasing the partitioning kafka topics in general to distribute the load
> better and increase parallelism.
>
> then i try to recreate my kafka topic to 4 partition because i have 4
> datanode on my elastic search. then, i check the broker node in zookeeper
> for all the topics that i create and i've seen there are *[1,2,3]*
> partition inside the */brokers/topics/(all_my_topics_name)/partitions*
> But when i start the storm topology, i've seen this error message on my
> storm logs if there is no node for
>
> */brokers/topics/(all_my_topics_name)/partitions/0/state *Why it still
> read the *0 *partitions.? is the metron just set for assigned 1 partition
> only on kafka topic?
> then i try to create manually the */brokers/topics/(all_my_topics_name)/partitions/0/state
> *but i've found another error msg like this:
>
>
>
> any suggestion about this?
>
>
>
>
> --
>
> Jon
>
>
> --
Jon