You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Christian Tramnitz <tr...@trasec.de> on 2017/05/10 13:56:05 UTC
ES silently stops indexing after 10k messages
Is anyone aware of a config setting that could cause the indexing topology to stop after writing 10k messages?
I have a parser running, writing into the indexing topic. Upon restart, the indexing topology (just ES enabled for now) picks up the latest (not oldest non-committed!) messages and puts 10000 into the current index. Then it silently stops. No logs (ES, Kafka, Storm) give an indication what’s preventing it from going on. When restarting the topology, it does another round of 10k messages and stops again. Very weird.
This is on a cluster build from Friday’s master and bare metal installation according to the documentation.
Thanks,
Christian
Re: ES silently stops indexing after 10k messages
Posted by Christian Tramnitz <tr...@trasec.de>.
Hi Ryan,
great pointer, I just ended up with the same conclusion. Increasing Storm debug level to “DEBUG” I can see offset problems like:
2017-05-10 15:28:36.332 o.a.s.k.s.KafkaSpout Thread-13-kafkaSpout-executor[5 5] [DEBUG] Unexpected offset found [342356]. OffsetEntry{topic-partition=indexing-0, fetchOffset=2988315, committedOffset=2988314, ackedMsgs=[{topic-partition=indexing-0, offset=342356, numFails=0}, {topic-partition=indexing-0, off
set=342357, numFails=0}
I have seen that before on another installation, but only while the backlog was growing into Kafka retention time and Kafka purged older messages before they could be consumed.
But here the retention is configured to 10GB while only 5.4G reside in the topic.
I have restarted the topology with offset strategy LATEST (instead of UNCOMMITTED_EARLIEST) now and don’t run into the 10k msg limit anymore. I’ll do a little more Kafka debugging and probably switching back to UNCOMMITTED_EARLIEST to see if the issue is resolved now. If not, my first guess would be that the spout cannot write the offset back and tries to start over again. I’ve seen a similar (non-Metron related) problem on the Storm ML a few weeks ago. If it is resolved, then it just failed once (or the topic was broken as you mentioned) and running with LATEST fixed it by writing a more recent (and available) offset.
BR,
Christian
Von: Ryan Merriman <me...@gmail.com>
Antworten an: "user@metron.apache.org" <us...@metron.apache.org>
Datum: Mittwoch, 10. Mai 2017 um 17:14
An: "user@metron.apache.org" <us...@metron.apache.org>
Betreff: Re: ES silently stops indexing after 10k messages
Christian,
We happened to run into this exact situation a couple days ago while tuning the indexing topology. In our case we were testing HDFS write performance and ES wasn't involved at all. We eventually tracked it down to a bad topic and we suspect the offsets were messed up somehow. Creating a new topic resolved the problem. Not sure what the root cause is at the moment but hopefully this workaround can get you past it. Hope this helps.
Ryan
On Wed, May 10, 2017 at 8:56 AM, Christian Tramnitz <tr...@trasec.de>> wrote:
Is anyone aware of a config setting that could cause the indexing topology to stop after writing 10k messages?
I have a parser running, writing into the indexing topic. Upon restart, the indexing topology (just ES enabled for now) picks up the latest (not oldest non-committed!) messages and puts 10000 into the current index. Then it silently stops. No logs (ES, Kafka, Storm) give an indication what’s preventing it from going on. When restarting the topology, it does another round of 10k messages and stops again. Very weird.
This is on a cluster build from Friday’s master and bare metal installation according to the documentation.
Thanks,
Christian
Re: ES silently stops indexing after 10k messages
Posted by Ryan Merriman <me...@gmail.com>.
Christian,
We happened to run into this exact situation a couple days ago while tuning
the indexing topology. In our case we were testing HDFS write performance
and ES wasn't involved at all. We eventually tracked it down to a bad
topic and we suspect the offsets were messed up somehow. Creating a new
topic resolved the problem. Not sure what the root cause is at the moment
but hopefully this workaround can get you past it. Hope this helps.
Ryan
On Wed, May 10, 2017 at 8:56 AM, Christian Tramnitz <tr...@trasec.de>
wrote:
> Is anyone aware of a config setting that could cause the indexing topology
> to stop after writing 10k messages?
>
>
>
> I have a parser running, writing into the indexing topic. Upon restart,
> the indexing topology (just ES enabled for now) picks up the latest (not
> oldest non-committed!) messages and puts 10000 into the current index. Then
> it silently stops. No logs (ES, Kafka, Storm) give an indication what’s
> preventing it from going on. When restarting the topology, it does another
> round of 10k messages and stops again. Very weird.
>
>
>
> This is on a cluster build from Friday’s master and bare metal
> installation according to the documentation.
>
>
>
>
>
> Thanks,
>
> Christian
>