You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Julien Nioche <li...@gmail.com> on 2014/07/21 12:45:45 UTC
Storm UI : handle custom stream as a system one
Hi,
I have a custom stream for handling logs (called '_log') and send them to
ElasticSearch for indexing. The log tuples are generated by my spouts and
bolts. My pipeline also uses the default stream for the normal processing
of tuples from RabbitMQ.
Everything works fine but I would like to be able to treat this _log stream
as one of the system ones (e.g. __metrics) and be able to hide them from
the stats. The summary of Emitted / Transferred currently takes these log
events into account which is not very useful.
I tried renaming the stream into '__log' but this resulted in an error when
trying to start the topoloy.
Any idea of how I could do that?
Thanks
Julien
--
Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Storm UI : handle custom stream as a system one
Posted by Julien Nioche <li...@gmail.com>.
Thanks Harsha. I created https://issues.apache.org/jira/browse/STORM-416
On 21 July 2014 17:05, Harsha <st...@harsha.io> wrote:
> the code is in core.clj in mk-include-sys-fn which calls
> system-id?(common.clj). I think UI code is fine but nimbus won't accept a
> topology which has stream-id with "__" at the beginning. We can probably
> add a exception in nimbus if a stream starts with "__" and its in
> storm.user.system.streams. Can you please file a jira to track this.
> Thanks.
> -Harsha
>
>
> On Mon, Jul 21, 2014, at 08:59 AM, Julien Nioche wrote:
>
> Yes, that's also the conclusion I came to.
> I could not find where in the UI code is the call to Utils.isSystemId(String).
> One option would be to be able to define in the configuration a list of
> streams to treat as system. Does the UI code access the configuration files
> at all?
>
> Thanks Harsha
>
>
> On 21 July 2014 16:43, Harsha <st...@harsha.io> wrote:
>
>
> thats caused by validate-ids! function which checks if the users stream id
> is system id and throws that exception.
> So looks like "__" reserved for system streams only not allowed for users.
>
>
> On Mon, Jul 21, 2014, at 08:30 AM, Julien Nioche wrote:
>
> Hi Harsha
>
> Am getting :
>
> 5935 [main] WARN backtype.storm.daemon.nimbus - Topology submission
> exception. (topology name='QueuePopulator') #<InvalidTopologyException
> InvalidTopologyException(msg:__log is not a valid stream id)>
> 5941 [main] ERROR org.apache.zookeeper.server.NIOServerCnxnFactory -
> Thread Thread[main,5,main] died
> backtype.storm.generated.InvalidTopologyException: null
> at backtype.storm.daemon.common$validate_ids_BANG_.invoke(common.clj:126)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.common$validate_basic_BANG_.invoke(common.clj:142)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.common$system_topology_BANG_.invoke(common.clj:297)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> Thanks
>
> Julien
>
>
> On 21 July 2014 16:22, Harsha <st...@harsha.io> wrote:
>
>
> Hi Julien,
> UI code calls Utils.isSystemId(String) which checks if the stream
> id starts with "__". What error are you seeing when you renamed into
> "__log".
> -Harsha
>
>
>
> On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote:
>
> Hi,
>
> I have a custom stream for handling logs (called '_log') and send them to
> ElasticSearch for indexing. The log tuples are generated by my spouts and
> bolts. My pipeline also uses the default stream for the normal processing
> of tuples from RabbitMQ.
>
> Everything works fine but I would like to be able to treat this _log
> stream as one of the system ones (e.g. __metrics) and be able to hide
> them from the stats. The summary of Emitted / Transferred currently takes
> these log events into account which is not very useful.
>
> I tried renaming the stream into '__log' but this resulted in an error
> when trying to start the topoloy.
>
> Any idea of how I could do that?
>
> Thanks
>
> Julien
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
>
>
>
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
>
>
>
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
>
--
Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Storm UI : handle custom stream as a system one
Posted by Harsha <st...@harsha.io>.
the code is in core.clj in mk-include-sys-fn which calls
system-id?(common.clj). I think UI code is fine but nimbus
won't accept a topology which has stream-id with "__" at the
beginning. We can probably add a exception in nimbus if a
stream starts with "__" and its in storm.user.system.streams.
Can you please file a jira to track this. Thanks.
-Harsha
On Mon, Jul 21, 2014, at 08:59 AM, Julien Nioche wrote:
Yes, that's also the conclusion I came to.
I could not find where in the UI code is the call
to Utils.isSystemId(String). One option would be to be able to
define in the configuration a list of streams to treat as
system. Does the UI code access the configuration files at all?
Thanks Harsha
On 21 July 2014 16:43, Harsha <[1...@harsha.io> wrote:
thats caused by validate-ids! function which checks if the
users stream id is system id and throws that exception.
So looks like "__" reserved for system streams only not allowed
for users.
On Mon, Jul 21, 2014, at 08:30 AM, Julien Nioche wrote:
Hi Harsha
Am getting :
5935 [main] WARN backtype.storm.daemon.nimbus - Topology
submission exception. (topology name='QueuePopulator')
#<InvalidTopologyException InvalidTopologyException(msg:__log
is not a valid stream id)>
5941 [main] ERROR
org.apache.zookeeper.server.NIOServerCnxnFactory - Thread
Thread[main,5,main] died
backtype.storm.generated.InvalidTopologyException: null
at
backtype.storm.daemon.common$validate_ids_BANG_.invoke(common.c
lj:126) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.common$validate_basic_BANG_.invoke(common
.clj:142) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.common$system_topology_BANG_.invoke(commo
n.clj:297) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
Thanks
Julien
On 21 July 2014 16:22, Harsha <[2...@harsha.io> wrote:
Hi Julien,
UI code calls Utils.isSystemId(String) which checks if
the stream id starts with "__". What error are you seeing when
you renamed into "__log".
-Harsha
On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote:
Hi,
I have a custom stream for handling logs (called '_log') and
send them to ElasticSearch for indexing. The log tuples are
generated by my spouts and bolts. My pipeline also uses the
default stream for the normal processing of tuples from
RabbitMQ.
Everything works fine but I would like to be able to treat this
_log stream as one of the system ones (e.g. __metrics) and be
able to hide them from the stats. The summary of Emitted /
Transferred currently takes these log events into account which
is not very useful.
I tried renaming the stream into '__log' but this resulted in
an error when trying to start the topoloy.
Any idea of how I could do that?
Thanks
Julien
--
[logo.gif]
Open Source Solutions for Text Engineering
[3]http://digitalpebble.blogspot.com/
[4]http://www.digitalpebble.com
[5]http://twitter.com/digitalpebble
--
[logo.gif]
Open Source Solutions for Text Engineering
[6]http://digitalpebble.blogspot.com/
[7]http://www.digitalpebble.com
[8]http://twitter.com/digitalpebble
--
[logo.gif]
Open Source Solutions for Text Engineering
[9]http://digitalpebble.blogspot.com/
[10]http://www.digitalpebble.com
[11]http://twitter.com/digitalpebble
References
1. mailto:storm@harsha.io
2. mailto:storm@harsha.io
3. http://digitalpebble.blogspot.com/
4. http://www.digitalpebble.com/
5. http://twitter.com/digitalpebble
6. http://digitalpebble.blogspot.com/
7. http://www.digitalpebble.com/
8. http://twitter.com/digitalpebble
9. http://digitalpebble.blogspot.com/
10. http://www.digitalpebble.com/
11. http://twitter.com/digitalpebble
Re: Storm UI : handle custom stream as a system one
Posted by Julien Nioche <li...@gmail.com>.
Yes, that's also the conclusion I came to.
I could not find where in the UI code is the call to Utils.isSystemId(String).
One option would be to be able to define in the configuration a list of
streams to treat as system. Does the UI code access the configuration files
at all?
Thanks Harsha
On 21 July 2014 16:43, Harsha <st...@harsha.io> wrote:
> thats caused by validate-ids! function which checks if the users stream
> id is system id and throws that exception.
> So looks like "__" reserved for system streams only not allowed for users.
>
>
> On Mon, Jul 21, 2014, at 08:30 AM, Julien Nioche wrote:
>
> Hi Harsha
>
> Am getting :
>
> 5935 [main] WARN backtype.storm.daemon.nimbus - Topology submission
> exception. (topology name='QueuePopulator') #<InvalidTopologyException
> InvalidTopologyException(msg:__log is not a valid stream id)>
> 5941 [main] ERROR org.apache.zookeeper.server.NIOServerCnxnFactory -
> Thread Thread[main,5,main] died
> backtype.storm.generated.InvalidTopologyException: null
> at backtype.storm.daemon.common$validate_ids_BANG_.invoke(common.clj:126)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.common$validate_basic_BANG_.invoke(common.clj:142)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.common$system_topology_BANG_.invoke(common.clj:297)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> Thanks
>
> Julien
>
>
> On 21 July 2014 16:22, Harsha <st...@harsha.io> wrote:
>
>
> Hi Julien,
> UI code calls Utils.isSystemId(String) which checks if the stream
> id starts with "__". What error are you seeing when you renamed into
> "__log".
> -Harsha
>
>
>
> On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote:
>
> Hi,
>
> I have a custom stream for handling logs (called '_log') and send them to
> ElasticSearch for indexing. The log tuples are generated by my spouts and
> bolts. My pipeline also uses the default stream for the normal processing
> of tuples from RabbitMQ.
>
> Everything works fine but I would like to be able to treat this _log
> stream as one of the system ones (e.g. __metrics) and be able to hide
> them from the stats. The summary of Emitted / Transferred currently takes
> these log events into account which is not very useful.
>
> I tried renaming the stream into '__log' but this resulted in an error
> when trying to start the topoloy.
>
> Any idea of how I could do that?
>
> Thanks
>
> Julien
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
>
>
>
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
>
--
Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Storm UI : handle custom stream as a system one
Posted by Harsha <st...@harsha.io>.
thats caused by validate-ids! function which checks if the
users stream id is system id and throws that exception.
So looks like "__" reserved for system streams only not allowed
for users.
On Mon, Jul 21, 2014, at 08:30 AM, Julien Nioche wrote:
Hi Harsha
Am getting :
5935 [main] WARN backtype.storm.daemon.nimbus - Topology
submission exception. (topology name='QueuePopulator')
#<InvalidTopologyException InvalidTopologyException(msg:__log
is not a valid stream id)>
5941 [main] ERROR
org.apache.zookeeper.server.NIOServerCnxnFactory - Thread
Thread[main,5,main] died
backtype.storm.generated.InvalidTopologyException: null
at
backtype.storm.daemon.common$validate_ids_BANG_.invoke(common.c
lj:126) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.common$validate_basic_BANG_.invoke(common
.clj:142) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.common$system_topology_BANG_.invoke(commo
n.clj:297) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
Thanks
Julien
On 21 July 2014 16:22, Harsha <[1...@harsha.io> wrote:
Hi Julien,
UI code calls Utils.isSystemId(String) which checks if
the stream id starts with "__". What error are you seeing when
you renamed into "__log".
-Harsha
On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote:
Hi,
I have a custom stream for handling logs (called '_log') and
send them to ElasticSearch for indexing. The log tuples are
generated by my spouts and bolts. My pipeline also uses the
default stream for the normal processing of tuples from
RabbitMQ.
Everything works fine but I would like to be able to treat this
_log stream as one of the system ones (e.g. __metrics) and be
able to hide them from the stats. The summary of Emitted /
Transferred currently takes these log events into account which
is not very useful.
I tried renaming the stream into '__log' but this resulted in
an error when trying to start the topoloy.
Any idea of how I could do that?
Thanks
Julien
--
[logo.gif]
Open Source Solutions for Text Engineering
[2]http://digitalpebble.blogspot.com/
[3]http://www.digitalpebble.com
[4]http://twitter.com/digitalpebble
--
[logo.gif]
Open Source Solutions for Text Engineering
[5]http://digitalpebble.blogspot.com/
[6]http://www.digitalpebble.com
[7]http://twitter.com/digitalpebble
References
1. mailto:storm@harsha.io
2. http://digitalpebble.blogspot.com/
3. http://www.digitalpebble.com/
4. http://twitter.com/digitalpebble
5. http://digitalpebble.blogspot.com/
6. http://www.digitalpebble.com/
7. http://twitter.com/digitalpebble
Re: Storm UI : handle custom stream as a system one
Posted by Julien Nioche <li...@gmail.com>.
Hi Harsha
Am getting :
5935 [main] WARN backtype.storm.daemon.nimbus - Topology submission
exception. (topology name='QueuePopulator') #<InvalidTopologyException
InvalidTopologyException(msg:__log is not a valid stream id)>
5941 [main] ERROR org.apache.zookeeper.server.NIOServerCnxnFactory - Thread
Thread[main,5,main] died
backtype.storm.generated.InvalidTopologyException: null
at backtype.storm.daemon.common$validate_ids_BANG_.invoke(common.clj:126)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.common$validate_basic_BANG_.invoke(common.clj:142)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.common$system_topology_BANG_.invoke(common.clj:297)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
Thanks
Julien
On 21 July 2014 16:22, Harsha <st...@harsha.io> wrote:
> Hi Julien,
> UI code calls Utils.isSystemId(String) which checks if the stream
> id starts with "__". What error are you seeing when you renamed into
> "__log".
> -Harsha
>
>
> On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote:
>
> Hi,
>
> I have a custom stream for handling logs (called '_log') and send them to
> ElasticSearch for indexing. The log tuples are generated by my spouts and
> bolts. My pipeline also uses the default stream for the normal processing
> of tuples from RabbitMQ.
>
> Everything works fine but I would like to be able to treat this _log
> stream as one of the system ones (e.g. __metrics) and be able to hide
> them from the stats. The summary of Emitted / Transferred currently takes
> these log events into account which is not very useful.
>
> I tried renaming the stream into '__log' but this resulted in an error
> when trying to start the topoloy.
>
> Any idea of how I could do that?
>
> Thanks
>
> Julien
>
> --
>
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
>
--
Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Storm UI : handle custom stream as a system one
Posted by Harsha <st...@harsha.io>.
Hi Julien,
UI code calls Utils.isSystemId(String) which checks if
the stream id starts with "__". What error are you seeing when
you renamed into "__log".
-Harsha
On Mon, Jul 21, 2014, at 03:45 AM, Julien Nioche wrote:
Hi,
I have a custom stream for handling logs (called '_log') and
send them to ElasticSearch for indexing. The log tuples are
generated by my spouts and bolts. My pipeline also uses the
default stream for the normal processing of tuples from
RabbitMQ.
Everything works fine but I would like to be able to treat this
_log stream as one of the system ones (e.g. __metrics) and be
able to hide them from the stats. The summary of Emitted /
Transferred currently takes these log events into account which
is not very useful.
I tried renaming the stream into '__log' but this resulted in
an error when trying to start the topoloy.
Any idea of how I could do that?
Thanks
Julien
--
[logo.gif]
Open Source Solutions for Text Engineering
[1]http://digitalpebble.blogspot.com/
[2]http://www.digitalpebble.com
[3]http://twitter.com/digitalpebble
References
1. http://digitalpebble.blogspot.com/
2. http://www.digitalpebble.com/
3. http://twitter.com/digitalpebble
RE: Storm UI : handle custom stream as a system one
Posted by Ross Mills <rm...@actify.com>.
unsubscribe
From: Julien Nioche [mailto:lists.digitalpebble@gmail.com]
Sent: 21 July 2014 11:46
To: user@storm.incubator.apache.org
Subject: Storm UI : handle custom stream as a system one
Hi,
I have a custom stream for handling logs (called '_log') and send them to ElasticSearch for indexing. The log tuples are generated by my spouts and bolts. My pipeline also uses the default stream for the normal processing of tuples from RabbitMQ.
Everything works fine but I would like to be able to treat this _log stream as one of the system ones (e.g. __metrics) and be able to hide them from the stats. The summary of Emitted / Transferred currently takes these log events into account which is not very useful.
I tried renaming the stream into '__log' but this resulted in an error when trying to start the topoloy.
Any idea of how I could do that?
Thanks
Julien
--
[Image removed by sender.]
Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble