You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/06/10 19:29:36 UTC
[GitHub] incubator-metron pull request #149: METRON-217: Found a grabbag of bugs
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/149
METRON-217: Found a grabbag of bugs
Doing some scale testing, I found a grabbag of bugs:
* The elasticsearch writer should allow multiple ES hosts to be passed in via a List. This should be backwards compatible.
* There is a concurrent modification exception while cloning the message field
* The parser topology numSpoutTasks and numParserTasks are swapped
* The grok parser should throw an exception if it cannot parse so the message that could not be parsed is sent to the error queue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron bug_squash_1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/149.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #149
----
commit 3738df5112bf7514fb1a9e9b6c7d812ce97a8a69
Author: cstella <ce...@gmail.com>
Date: 2016-06-10T19:17:57Z
Fixing a grabbag of bugs.
commit 08609a91483683c241e7f3e706630f79e25bffcb
Author: cstella <ce...@gmail.com>
Date: 2016-06-10T19:28:11Z
Updating writer bolt.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/149
Overall ran well. Increasing the indexWriter executors to 2 didn't create the ConcurrentModificationException. Increasing the kafkaSpout executors to 2 gave me the following exception, but the topology recovered.
I'm +1 on the patch, but I've left my test rig up in case you wish to do some additional debugging.
Trace:
2016-06-10 22:23:36.438 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: java.io.OptionalDataException
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$fn__5495$fn__5508$fn__5559.invoke(executor.clj:808) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.lang.RuntimeException: java.io.OptionalDataException
at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:?]
at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
Caused by: java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1371) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[?:1.8.0_40]
at java.util.HashMap.readObject(HashMap.java:1394) ~[?:1.8.0_40]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_40]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_40]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[?:1.8.0_40]
at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:?]
at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
2016-06-10 22:23:36.441 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.OptionalDataException
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$fn__5495$fn__5508$fn__5559.invoke(executor.clj:808) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.lang.RuntimeException: java.io.OptionalDataException
at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:?]
at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
Caused by: java.io.OptionalDataException
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1371) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[?:1.8.0_40]
at java.util.HashMap.readObject(HashMap.java:1394) ~[?:1.8.0_40]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_40]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_40]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) ~[?:1.8.0_40]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[?:1.8.0_40]
at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:?]
at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:423) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.disruptor$clojure_handler$reify__4994.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
... 6 more
2016-06-10 22:23:36.467 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:332) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
at backtype.storm.daemon.worker$fn__6068$fn__6069.invoke(worker.clj:636) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.daemon.executor$mk_executor_data$fn__5328$fn__5329.invoke(executor.clj:256) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:485) [storm-core-0.10.0.2.3.4.7-4.jar:0.10.0.2.3.4.7-4]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/149
+ 1from me. great job
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/149
@dlyle65535 any chance you can run that again with this commit?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #149: METRON-217: Found a grabbag of bugs
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/149
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/149
+1 from me pending EC2 test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/149
Just for those poor souls who are debugging this on the internet, this is the sister exception (from the other thread):
> 2016-06-10 13:34:44.687 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.util.ConcurrentModificationException
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.daemon.executor$fn__5492$fn__5505$fn__5556.invoke(executor.clj:813) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.util$async_loop$fn__644.invoke(util.clj:479) [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429) ~[?:1.8.0_51]
at java.util.HashMap$EntryIterator.next(HashMap.java:1463) ~[?:1.8.0_51]
at java.util.HashMap$EntryIterator.next(HashMap.java:1461) ~[?:1.8.0_51]
at java.util.HashMap.putMapEntries(HashMap.java:511) ~[?:1.8.0_51]
at java.util.HashMap.clone(HashMap.java:1324) ~[?:1.8.0_51]
at org.apache.metron.enrichment.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:66) ~[stormjar.jar:?]
at backtype.storm.daemon.executor$fn__5492$tuple_action_fn__5494.invoke(executor.clj:684) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5415.invoke(executor.clj:431) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.disruptor$clojure_handler$reify__4991.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
... 6 more
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/149
I also went ahead and added a bit better logging if the CME returns, we should get a sense of what the other threads are doing. Definitely would like to see if you see any CMEs with the current commit. If so, it'd be great to check the error log, the full thread dump for that process should be there at the point of the CME.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/149
So, that's the other side of the concurrent modification exception, @dlyle65535. Somehow we are modifying the JSONObject message while it's deserializing, like in https://groups.google.com/forum/#!topic/storm-user/67ycy_WLqEQ except while deserializing, not serializing.
This exception is the CME from the deserialization code's view. The trick here is that I'm not sure how it's deserializing while we're using it. I'll have to consider this more.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---