You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Jeremiah Adams <JA...@helixeducation.com> on 2018/06/05 13:55:37 UTC
MessageCollector and WindowableTask
From th?e docs:
"If you need to send messages to output streams, you can use the MessageCollector<http://samza.apache.org/learn/documentation/0.14/api/javadocs/org/apache/samza/task/MessageCollector.html> object passed to the window() method. Please only use that MessageCollector object for sending messages, and don't use it outside of the call to window()."
Why MessageCollector should not be used outside the call to window() in a WindowableTask?
Does this mean I should not pass MessageCollector around as a method argument to child objects responsible for processing the window()?
Jeremiah Adams
Software Engineer
www.helixeducation.com<http://www.helixeducation.com/>
Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> | LinkedIn<http://www.linkedin.com/company/3609946>
Re: MessageCollector and WindowableTask
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Hey Jeremiah,
Here's a standard pattern that Samza users in the community have adopted.
Configure your job to consume from your input-topic that contains a stream
of requests.
In your process() call:
- If you need to retry your request, store it for later processing in a
RocksDB store
- Configure a change-log for the store so that you get recoverability
In your window() call:
- Scan each entry in your store and re-attempt your remote request
- If the remote request succeeds, delete the entry from your store
Please let me know if you had further questions.
On Tue, Jun 5, 2018 at 10:16 AM, Jeremiah Adams <JA...@helixeducation.com>
wrote:
>
> I was hoping I could use Windowing to process a topic every n-seconds for
> some lengthy retry logic. E.g. some tasks can be retried n-times every
> m-seconds if they fail. Failed attempts are re-queued for n-attempts before
> being considered a real failure. e.g. POSTing to an endpoint fails due to
> outage - so retry in the event the network connectivity can be restored.
>
> My ideas so far:
> 1. monitor two topics - one is attempt, two is reattempt. Process these
> separately with distinct Tasks. Re-queue the re-attempts and process them
> as the appropriate timespan expires. Wasteful concerning processing/network
> resources in doing so. The task will spend a ton of time just requeuing
> waiting for the appropriate timespan.
> 2. upon process() check rocksDB for reattemtps. Reprocess if so, then
> process the original IncommingMessageEnvolope. Feels contrived - i'm
> processing reattempts as a side-effect of processing a normal message.
>
> Keep in mind our data is nowhere near 'big'.
>
> Jeremiah Adams
> Software Engineer
> www.helixeducation.com
> Blog | Twitter | Facebook | LinkedIn
>
> ________________________________________
> From: Jagadish Venkatraman <ja...@gmail.com>
> Sent: Tuesday, June 5, 2018 10:22 AM
> To: dev@samza.apache.org
> Subject: Re: MessageCollector and WindowableTask
>
> Hey Jeremiah,
>
> >> Please only use that MessageCollector object for sending messages, and
> don't use it outside of the call to window(). Why MessageCollector should
> not be used outside the call to window() in a WindowableTask?
>
> The collector and coordinator instances are expected to be used within a
> context of a single window call. All this means is that a window should not
> make any assumptions that the same instance of the collector will be
> provided across invocations.
>
> >> Does this mean I should not pass MessageCollector around as a method
> argument to child objects responsible for processing the window()?
>
> As long as those objects/helper methods are scoped within a single window
> call, it should be fine.
>
> Can you please provide some background on what you are trying to do?
>
> Thanks,
> Jagadish
>
>
>
> On Tue, Jun 5, 2018 at 6:55 AM, Jeremiah Adams <JA...@helixeducation.com>
> wrote:
>
> > From th?e docs:
> >
> >
> > "If you need to send messages to output streams, you can use the
> > MessageCollector<https://url.emailprotection.link/?abxQT9w4aKBPgxuwj-
> FWE3xNwSOvLPAXEqUg1ZOXefiB62v9ush8qrBPEEYxuqiJ-TKVEuQiECD5ysLF4m1RuXA~~
> > javadocs/org/apache/samza/task/MessageCollector.html> object passed to
> > the window() method. Please only use that MessageCollector object for
> > sending messages, and don't use it outside of the call to window()."
> >
> >
> > Why MessageCollector should not be used outside the call to window() in a
> > WindowableTask?
> >
> > Does this mean I should not pass MessageCollector around as a method
> > argument to child objects responsible for processing the window()?
> >
> >
> > Jeremiah Adams
> > Software Engineer
> > https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfI
> erU3ZwsA3Gv_C0~<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-
> qKE3Xn2gNiZ3dlqAeSDA~>
> > Blog<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-
> KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~> | Twitter<
> https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zwnjt5JsoFoNU2KX2
> AeBPVE~
> > HelixEducation> | Facebook<https://url.emailprotection.link/?aAmyAO_
> nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV
> -dMSER5wl8zDVRr3RsS0OmcF9A~~> |
> > LinkedIn<https://url.emailprotection.link/?aanlcNI-cN74Gdz-
> TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~>
> >
>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: MessageCollector and WindowableTask
Posted by Jeremiah Adams <JA...@helixeducation.com>.
I was hoping I could use Windowing to process a topic every n-seconds for some lengthy retry logic. E.g. some tasks can be retried n-times every m-seconds if they fail. Failed attempts are re-queued for n-attempts before being considered a real failure. e.g. POSTing to an endpoint fails due to outage - so retry in the event the network connectivity can be restored.
My ideas so far:
1. monitor two topics - one is attempt, two is reattempt. Process these separately with distinct Tasks. Re-queue the re-attempts and process them as the appropriate timespan expires. Wasteful concerning processing/network resources in doing so. The task will spend a ton of time just requeuing waiting for the appropriate timespan.
2. upon process() check rocksDB for reattemtps. Reprocess if so, then process the original IncommingMessageEnvolope. Feels contrived - i'm processing reattempts as a side-effect of processing a normal message.
Keep in mind our data is nowhere near 'big'.
Jeremiah Adams
Software Engineer
www.helixeducation.com
Blog | Twitter | Facebook | LinkedIn
________________________________________
From: Jagadish Venkatraman <ja...@gmail.com>
Sent: Tuesday, June 5, 2018 10:22 AM
To: dev@samza.apache.org
Subject: Re: MessageCollector and WindowableTask
Hey Jeremiah,
>> Please only use that MessageCollector object for sending messages, and
don't use it outside of the call to window(). Why MessageCollector should
not be used outside the call to window() in a WindowableTask?
The collector and coordinator instances are expected to be used within a
context of a single window call. All this means is that a window should not
make any assumptions that the same instance of the collector will be
provided across invocations.
>> Does this mean I should not pass MessageCollector around as a method
argument to child objects responsible for processing the window()?
As long as those objects/helper methods are scoped within a single window
call, it should be fine.
Can you please provide some background on what you are trying to do?
Thanks,
Jagadish
On Tue, Jun 5, 2018 at 6:55 AM, Jeremiah Adams <JA...@helixeducation.com>
wrote:
> From th?e docs:
>
>
> "If you need to send messages to output streams, you can use the
> MessageCollector<https://url.emailprotection.link/?abxQT9w4aKBPgxuwj-FWE3xNwSOvLPAXEqUg1ZOXefiB62v9ush8qrBPEEYxuqiJ-TKVEuQiECD5ysLF4m1RuXA~~
> javadocs/org/apache/samza/task/MessageCollector.html> object passed to
> the window() method. Please only use that MessageCollector object for
> sending messages, and don't use it outside of the call to window()."
>
>
> Why MessageCollector should not be used outside the call to window() in a
> WindowableTask?
>
> Does this mean I should not pass MessageCollector around as a method
> argument to child objects responsible for processing the window()?
>
>
> Jeremiah Adams
> Software Engineer
> https://url.emailprotection.link/?ahfhEufaAWbezBrUFPG98ZJcterGfIerU3ZwsA3Gv_C0~<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHp-qKE3Xn2gNiZ3dlqAeSDA~>
> Blog<https://url.emailprotection.link/?a49H2rNGIIBtQOw6md8OcHgFEZu-KYuiu8doY66NWwmmyWxz7kC-27Yfnbdgd2wyh5gjXUa6LMT_NRXsj1g1VVg~~> | Twitter<https://url.emailprotection.link/?a0Q7ct5_6cOdbJ86kpWB0zwnjt5JsoFoNU2KX2AeBPVE~
> HelixEducation> | Facebook<https://url.emailprotection.link/?aAmyAO_nS_C1aDgBLeKyGTu0tksTt1_mn2PcS8KJXNJPM04iRHKgX96qGgENV-dMSER5wl8zDVRr3RsS0OmcF9A~~> |
> LinkedIn<https://url.emailprotection.link/?aanlcNI-cN74Gdz-TD332xAl6lHu7TRNICWoHUFjYf-KlBjrCGHoYR65b3rl-OyW10nWFv6hwYvUSoVHL4b3vGA~~>
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: MessageCollector and WindowableTask
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Hey Jeremiah,
>> Please only use that MessageCollector object for sending messages, and
don't use it outside of the call to window(). Why MessageCollector should
not be used outside the call to window() in a WindowableTask?
The collector and coordinator instances are expected to be used within a
context of a single window call. All this means is that a window should not
make any assumptions that the same instance of the collector will be
provided across invocations.
>> Does this mean I should not pass MessageCollector around as a method
argument to child objects responsible for processing the window()?
As long as those objects/helper methods are scoped within a single window
call, it should be fine.
Can you please provide some background on what you are trying to do?
Thanks,
Jagadish
On Tue, Jun 5, 2018 at 6:55 AM, Jeremiah Adams <JA...@helixeducation.com>
wrote:
> From th?e docs:
>
>
> "If you need to send messages to output streams, you can use the
> MessageCollector<http://samza.apache.org/learn/documentation/0.14/api/
> javadocs/org/apache/samza/task/MessageCollector.html> object passed to
> the window() method. Please only use that MessageCollector object for
> sending messages, and don't use it outside of the call to window()."
>
>
> Why MessageCollector should not be used outside the call to window() in a
> WindowableTask?
>
> Does this mean I should not pass MessageCollector around as a method
> argument to child objects responsible for processing the window()?
>
>
> Jeremiah Adams
> Software Engineer
> www.helixeducation.com<http://www.helixeducation.com/>
> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/
> HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> |
> LinkedIn<http://www.linkedin.com/company/3609946>
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University