You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Marco Costantini <mc...@gmail.com> on 2017/11/21 17:31:00 UTC
Can I emit a Map?
I have a batching bolt which ultimately makes a Map<String, Set<String>>.
At a time I decide, I need to emit that map to the next bolt in the
topology. I've tried it and it works, but I'm curious as to whether this is
an anti-pattern or if there are problems with doing this that I am unaware
of.
Please and thanks,
Marco.
Re: Can I emit a Map?
Posted by Arun Mahadevan <ar...@apache.org>.
I think it would be fine as long as you ensure your map is Immutable or at-least its not changed after its constructed. I don’t think there would be any issues with sizes.
From: Toy <no...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Wednesday, November 22, 2017 at 2:50 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Can I emit a Map?
We also emit a Map and so far it's been fine. You can also encode the string if it gets too big but I would measure the performance first
On Tue, Nov 21, 2017 at 13:12 Mauro Giusti <ma...@microsoft.com> wrote:
We also emit a map, encoded in Json –
We don’t have any problem with that, but it is only a 28 records map and each entry has 5 records with 1 number, so it is kind of small.
If your map is very large, then you might want to consider transforming that into records and emit them for a next level bolt?
Mauro.
From: Hannum, Daniel [mailto:Daniel_Hannum@PremierInc.com]
Sent: Tuesday, November 21, 2017 12:48 PM
To: user@storm.apache.org
Subject: Re: Can I emit a Map?
I forget where I read it, but I was told that there is no de jure size limit on tuples. My topology will have tuples 1M in size, and outliers much larger than that. I haven’t seen any issues. Obviously at some point, you run out of memory…. ☺
From: Marco Costantini <mc...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, November 21, 2017 at 3:39 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Can I emit a Map?
****This email did not originate from the Premier, Inc. network. Use caution when opening attachments or clicking on URLs.*****
.
Thanks. What if it's big? My Strings end up being a couple thousand characters long, and each map has around 500 of them. Seems daunting to emit such a tuple. Any thoughts on that?
On Tue, Nov 21, 2017 at 7:23 PM, Stig Rohde Døssing <sr...@apache.org> wrote:
I think that as long as you ensure that everything in the map can be serialized, it's fine.
2017-11-21 18:31 GMT+01:00 Marco Costantini <mc...@gmail.com>:
I have a batching bolt which ultimately makes a Map<String, Set<String>>. At a time I decide, I need to emit that map to the next bolt in the topology. I've tried it and it works, but I'm curious as to whether this is an anti-pattern or if there are problems with doing this that I am unaware of.
Please and thanks,
Marco.
Re: Can I emit a Map?
Posted by Toy <no...@gmail.com>.
We also emit a Map and so far it's been fine. You can also encode the
string if it gets too big but I would measure the performance first
On Tue, Nov 21, 2017 at 13:12 Mauro Giusti <ma...@microsoft.com> wrote:
> We also emit a map, encoded in Json –
>
> We don’t have any problem with that, but it is only a 28 records map and
> each entry has 5 records with 1 number, so it is kind of small.
>
>
>
> If your map is very large, then you might want to consider transforming
> that into records and emit them for a next level bolt?
>
>
>
> Mauro.
>
>
>
> *From:* Hannum, Daniel [mailto:Daniel_Hannum@PremierInc.com]
> *Sent:* Tuesday, November 21, 2017 12:48 PM
> *To:* user@storm.apache.org
>
>
> *Subject:* Re: Can I emit a Map?
>
>
>
> I forget where I read it, but I was told that there is no de jure size
> limit on tuples. My topology will have tuples 1M in size, and outliers much
> larger than that. I haven’t seen any issues. Obviously at some point, you
> run out of memory…. ☺
>
>
>
> *From: *Marco Costantini <mc...@gmail.com>
> *Reply-To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Date: *Tuesday, November 21, 2017 at 3:39 PM
> *To: *"user@storm.apache.org" <us...@storm.apache.org>
> *Subject: *Re: Can I emit a Map?
>
>
>
> ****This email did not originate from the Premier, Inc. network. Use
> caution when opening attachments or clicking on URLs.*****
>
>
> .
>
> Thanks. What if it's big? My Strings end up being a couple thousand
> characters long, and each map has around 500 of them. Seems daunting to
> emit such a tuple. Any thoughts on that?
>
>
>
> On Tue, Nov 21, 2017 at 7:23 PM, Stig Rohde Døssing <sr...@apache.org>
> wrote:
>
> I think that as long as you ensure that everything in the map can be
> serialized, it's fine.
>
>
>
> 2017-11-21 18:31 GMT+01:00 Marco Costantini <mc...@gmail.com>:
>
> I have a batching bolt which ultimately makes a Map<String, Set<String>>.
> At a time I decide, I need to emit that map to the next bolt in the
> topology. I've tried it and it works, but I'm curious as to whether this is
> an anti-pattern or if there are problems with doing this that I am unaware
> of.
>
>
>
> Please and thanks,
>
> Marco.
>
>
>
>
>
RE: Can I emit a Map?
Posted by Mauro Giusti <ma...@microsoft.com>.
We also emit a map, encoded in Json –
We don’t have any problem with that, but it is only a 28 records map and each entry has 5 records with 1 number, so it is kind of small.
If your map is very large, then you might want to consider transforming that into records and emit them for a next level bolt?
Mauro.
From: Hannum, Daniel [mailto:Daniel_Hannum@PremierInc.com]
Sent: Tuesday, November 21, 2017 12:48 PM
To: user@storm.apache.org
Subject: Re: Can I emit a Map?
I forget where I read it, but I was told that there is no de jure size limit on tuples. My topology will have tuples 1M in size, and outliers much larger than that. I haven’t seen any issues. Obviously at some point, you run out of memory…. ☺
From: Marco Costantini <mc...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, November 21, 2017 at 3:39 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Can I emit a Map?
****This email did not originate from the Premier, Inc. network. Use caution when opening attachments or clicking on URLs.*****
.
Thanks. What if it's big? My Strings end up being a couple thousand characters long, and each map has around 500 of them. Seems daunting to emit such a tuple. Any thoughts on that?
On Tue, Nov 21, 2017 at 7:23 PM, Stig Rohde Døssing <sr...@apache.org>> wrote:
I think that as long as you ensure that everything in the map can be serialized, it's fine.
2017-11-21 18:31 GMT+01:00 Marco Costantini <mc...@gmail.com>>:
I have a batching bolt which ultimately makes a Map<String, Set<String>>. At a time I decide, I need to emit that map to the next bolt in the topology. I've tried it and it works, but I'm curious as to whether this is an anti-pattern or if there are problems with doing this that I am unaware of.
Please and thanks,
Marco.
Re: Can I emit a Map?
Posted by "Hannum, Daniel" <Da...@PremierInc.com>.
I forget where I read it, but I was told that there is no de jure size limit on tuples. My topology will have tuples 1M in size, and outliers much larger than that. I haven’t seen any issues. Obviously at some point, you run out of memory…. ☺
From: Marco Costantini <mc...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, November 21, 2017 at 3:39 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Can I emit a Map?
****This email did not originate from the Premier, Inc. network. Use caution when opening attachments or clicking on URLs.*****
.
Thanks. What if it's big? My Strings end up being a couple thousand characters long, and each map has around 500 of them. Seems daunting to emit such a tuple. Any thoughts on that?
On Tue, Nov 21, 2017 at 7:23 PM, Stig Rohde Døssing <sr...@apache.org>> wrote:
I think that as long as you ensure that everything in the map can be serialized, it's fine.
2017-11-21 18:31 GMT+01:00 Marco Costantini <mc...@gmail.com>>:
I have a batching bolt which ultimately makes a Map<String, Set<String>>. At a time I decide, I need to emit that map to the next bolt in the topology. I've tried it and it works, but I'm curious as to whether this is an anti-pattern or if there are problems with doing this that I am unaware of.
Please and thanks,
Marco.
Re: Can I emit a Map?
Posted by Marco Costantini <mc...@gmail.com>.
Thanks. What if it's big? My Strings end up being a couple thousand
characters long, and each map has around 500 of them. Seems daunting to
emit such a tuple. Any thoughts on that?
On Tue, Nov 21, 2017 at 7:23 PM, Stig Rohde Døssing <sr...@apache.org> wrote:
> I think that as long as you ensure that everything in the map can be
> serialized, it's fine.
>
> 2017-11-21 18:31 GMT+01:00 Marco Costantini <mc...@gmail.com>:
>
>> I have a batching bolt which ultimately makes a Map<String, Set<String>>.
>> At a time I decide, I need to emit that map to the next bolt in the
>> topology. I've tried it and it works, but I'm curious as to whether this is
>> an anti-pattern or if there are problems with doing this that I am unaware
>> of.
>>
>> Please and thanks,
>> Marco.
>>
>
>
Re: Can I emit a Map?
Posted by Stig Rohde Døssing <sr...@apache.org>.
I think that as long as you ensure that everything in the map can be
serialized, it's fine.
2017-11-21 18:31 GMT+01:00 Marco Costantini <mc...@gmail.com>:
> I have a batching bolt which ultimately makes a Map<String, Set<String>>.
> At a time I decide, I need to emit that map to the next bolt in the
> topology. I've tried it and it works, but I'm curious as to whether this is
> an anti-pattern or if there are problems with doing this that I am unaware
> of.
>
> Please and thanks,
> Marco.
>