You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Jeremiah Adams <JA...@helixeducation.com> on 2019/04/03 19:05:27 UTC
Context when Converting from Yarn Task to Standalone
LocalApplicationRunner
?I am working to move our code from Yarn based StreamTask to standalone StreamApplication via LocalTaskRunner.
I'm having some trouble understanding how to create/fetch a Context for use in the IntiableFunction.init() interface. My old code used context to get the store for LocalCacheManager initialization so I need this context. I see no interesting methods in StreamAplicationDescriptor.
I am launching my StreamApplication via a TaskRunner:
Where/how do i get a Samza.context.Context?
public static void main( String[] args )
{
CommandLine cmdLine = new CommandLine();
OptionSet options = cmdLine.parser().parse(args);
Config config = cmdLine.loadConfig(options);
InquirySubmissionApp app= new InquirySubmissionApp(config);
// Need to get a context here?
app.init();
LocalApplicationRunner localApplicationRunner = new LocalApplicationRunner(app, config);
localApplicationRunner.run();
localApplicationRunner.waitForFinish();
}?
Jeremiah Adams
Software Engineer
www.helixeducation.com<http://www.helixeducation.com/>
Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> | LinkedIn<http://www.linkedin.com/company/3609946>
Re: Context when Converting from Yarn Task to Standalone
LocalApplicationRunner
Posted by Jeremiah Adams <JA...@helixeducation.com>.
Thanks for the clarification. I completely missed that there LocalApplicationRunner could accept a TaskApplication. The javadocs are actually hard to find.
Jeremiah Adams
Software Engineer
www.helixeducation.com
Blog | Twitter | Facebook | LinkedIn
________________________________________
From: Cameron Lee <ca...@gmail.com>
Sent: Wednesday, April 3, 2019 4:42 PM
To: dev@samza.apache.org
Subject: Re: Context when Converting from Yarn Task to Standalone LocalApplicationRunner
Hi Jeremiah,
If you would like to continue to use the low-level API (i.e. StreamTask),
then you actually will want to use TaskApplication, not StreamApplication.
Sorry for the confusion regarding the naming of the classes. Take a look at
https://url.emailprotection.link/?bMpPl793BLAF99JGtib05IK-b5xqjhiYbgl09LPOL3g4Lvs2RXwUyn9O2MIjKna6h3K2xJhZj5BPLOmiKtBtJbst7c66tfpu7Pr4Xys7UEmmnEsktHGGkwMLzvLIhWs5G
for some more information about using the low-level API. You will still
pass your TaskApplication to the LocalApplicationRunner.
You do not need to explicitly call any "init" method in your code. That
method will get called for you, as long as you implement
InitableTask.init(...).
If you would like, as Yi suggested, you can use
ApplicationContainerContext/ApplicationTaskContext to manage your
LocalCacheManager. However, your existing "init" code should also continue
to work when you start to use TaskApplication.
If you would actually like to also migrate to using the high-level API,
then take a look at
https://url.emailprotection.link/?bH34maaCnmUSs992TWa-Lp0tq5R-XQh0P2igsH4rH9J2JTaEGdQV4VC-GU37hMoLVCxZqTihr0ngHRXF3icpAWNbwTO_xSe438qX6gu56sDrEuYTBWjopXuP19BT8nXKw.
Cameron
On Wed, Apr 3, 2019 at 2:52 PM Yi Pan <ni...@gmail.com> wrote:
> Hi, Jeremiah,
>
> In the new apis, you should be using ApplicationContainerContextFactory and
> ApplicationTaskContextFactory to instantiate context objects used in the
> whole container or in a task instance, respectively. The context factories
> should be implemented as dependencies injected to your implementation of
> StreamApplication. In your example, you should add the context factories in
> the InquirySubmissionApp:
> public InquirySubmissionApp implements StreamApplication {
> @Override
> public void describe(StreamApplicationDescriptor appDescriptor) {
> appDescriptor.withApplicationContainerContextFactory(new
> MyContainerContextFactory(...));
> appDescriptor.withApplicationTaskContextFactory(new
> MyTaskContextFactory(...));
> // user processing logic using MessageStream and the transform
> operators...
> }
> }
>
> Best,
>
> -Yi
>
> On Wed, Apr 3, 2019 at 12:05 PM Jeremiah Adams <JA...@helixeducation.com>
> wrote:
>
> > ?I am working to move our code from Yarn based StreamTask to standalone
> > StreamApplication via LocalTaskRunner.
> >
> >
> > I'm having some trouble understanding how to create/fetch a Context for
> > use in the IntiableFunction.init() interface. My old code used context to
> > get the store for LocalCacheManager initialization so I need this
> context.
> > I see no interesting methods in StreamAplicationDescriptor.
> >
> >
> > I am launching my StreamApplication via a TaskRunner:
> >
> >
> > Where/how do i get a Samza.context.Context?
> >
> >
> > public static void main( String[] args )
> > {
> >
> > CommandLine cmdLine = new CommandLine();
> > OptionSet options = cmdLine.parser().parse(args);
> > Config config = cmdLine.loadConfig(options);
> >
> > InquirySubmissionApp app= new InquirySubmissionApp(config);
> > // Need to get a context here?
> > app.init();
> > LocalApplicationRunner localApplicationRunner = new
> > LocalApplicationRunner(app, config);
> > localApplicationRunner.run();
> > localApplicationRunner.waitForFinish();
> > }?
> >
> >
> >
> >
> >
> >
> >
> >
> > Jeremiah Adams
> > Software Engineer
> > https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>
> > Blog<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~> | Twitter<
> > https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~> | Facebook<
> > https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~> | LinkedIn<
> > https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>
> >
>
Re: Context when Converting from Yarn Task to Standalone LocalApplicationRunner
Posted by Cameron Lee <ca...@gmail.com>.
Hi Jeremiah,
If you would like to continue to use the low-level API (i.e. StreamTask),
then you actually will want to use TaskApplication, not StreamApplication.
Sorry for the confusion regarding the naming of the classes. Take a look at
https://samza.apache.org/learn/documentation/latest/api/low-level-api.html
for some more information about using the low-level API. You will still
pass your TaskApplication to the LocalApplicationRunner.
You do not need to explicitly call any "init" method in your code. That
method will get called for you, as long as you implement
InitableTask.init(...).
If you would like, as Yi suggested, you can use
ApplicationContainerContext/ApplicationTaskContext to manage your
LocalCacheManager. However, your existing "init" code should also continue
to work when you start to use TaskApplication.
If you would actually like to also migrate to using the high-level API,
then take a look at
https://samza.apache.org/learn/documentation/latest/api/high-level-api.html.
Cameron
On Wed, Apr 3, 2019 at 2:52 PM Yi Pan <ni...@gmail.com> wrote:
> Hi, Jeremiah,
>
> In the new apis, you should be using ApplicationContainerContextFactory and
> ApplicationTaskContextFactory to instantiate context objects used in the
> whole container or in a task instance, respectively. The context factories
> should be implemented as dependencies injected to your implementation of
> StreamApplication. In your example, you should add the context factories in
> the InquirySubmissionApp:
> public InquirySubmissionApp implements StreamApplication {
> @Override
> public void describe(StreamApplicationDescriptor appDescriptor) {
> appDescriptor.withApplicationContainerContextFactory(new
> MyContainerContextFactory(...));
> appDescriptor.withApplicationTaskContextFactory(new
> MyTaskContextFactory(...));
> // user processing logic using MessageStream and the transform
> operators...
> }
> }
>
> Best,
>
> -Yi
>
> On Wed, Apr 3, 2019 at 12:05 PM Jeremiah Adams <JA...@helixeducation.com>
> wrote:
>
> > ?I am working to move our code from Yarn based StreamTask to standalone
> > StreamApplication via LocalTaskRunner.
> >
> >
> > I'm having some trouble understanding how to create/fetch a Context for
> > use in the IntiableFunction.init() interface. My old code used context to
> > get the store for LocalCacheManager initialization so I need this
> context.
> > I see no interesting methods in StreamAplicationDescriptor.
> >
> >
> > I am launching my StreamApplication via a TaskRunner:
> >
> >
> > Where/how do i get a Samza.context.Context?
> >
> >
> > public static void main( String[] args )
> > {
> >
> > CommandLine cmdLine = new CommandLine();
> > OptionSet options = cmdLine.parser().parse(args);
> > Config config = cmdLine.loadConfig(options);
> >
> > InquirySubmissionApp app= new InquirySubmissionApp(config);
> > // Need to get a context here?
> > app.init();
> > LocalApplicationRunner localApplicationRunner = new
> > LocalApplicationRunner(app, config);
> > localApplicationRunner.run();
> > localApplicationRunner.waitForFinish();
> > }?
> >
> >
> >
> >
> >
> >
> >
> >
> > Jeremiah Adams
> > Software Engineer
> > www.helixeducation.com<http://www.helixeducation.com/>
> > Blog<http://www.helixeducation.com/blog/> | Twitter<
> > https://twitter.com/HelixEducation> | Facebook<
> > https://www.facebook.com/HelixEducation> | LinkedIn<
> > http://www.linkedin.com/company/3609946>
> >
>
Re: Context when Converting from Yarn Task to Standalone LocalApplicationRunner
Posted by Yi Pan <ni...@gmail.com>.
Hi, Jeremiah,
In the new apis, you should be using ApplicationContainerContextFactory and
ApplicationTaskContextFactory to instantiate context objects used in the
whole container or in a task instance, respectively. The context factories
should be implemented as dependencies injected to your implementation of
StreamApplication. In your example, you should add the context factories in
the InquirySubmissionApp:
public InquirySubmissionApp implements StreamApplication {
@Override
public void describe(StreamApplicationDescriptor appDescriptor) {
appDescriptor.withApplicationContainerContextFactory(new
MyContainerContextFactory(...));
appDescriptor.withApplicationTaskContextFactory(new
MyTaskContextFactory(...));
// user processing logic using MessageStream and the transform
operators...
}
}
Best,
-Yi
On Wed, Apr 3, 2019 at 12:05 PM Jeremiah Adams <JA...@helixeducation.com>
wrote:
> ?I am working to move our code from Yarn based StreamTask to standalone
> StreamApplication via LocalTaskRunner.
>
>
> I'm having some trouble understanding how to create/fetch a Context for
> use in the IntiableFunction.init() interface. My old code used context to
> get the store for LocalCacheManager initialization so I need this context.
> I see no interesting methods in StreamAplicationDescriptor.
>
>
> I am launching my StreamApplication via a TaskRunner:
>
>
> Where/how do i get a Samza.context.Context?
>
>
> public static void main( String[] args )
> {
>
> CommandLine cmdLine = new CommandLine();
> OptionSet options = cmdLine.parser().parse(args);
> Config config = cmdLine.loadConfig(options);
>
> InquirySubmissionApp app= new InquirySubmissionApp(config);
> // Need to get a context here?
> app.init();
> LocalApplicationRunner localApplicationRunner = new
> LocalApplicationRunner(app, config);
> localApplicationRunner.run();
> localApplicationRunner.waitForFinish();
> }?
>
>
>
>
>
>
>
>
> Jeremiah Adams
> Software Engineer
> www.helixeducation.com<http://www.helixeducation.com/>
> Blog<http://www.helixeducation.com/blog/> | Twitter<
> https://twitter.com/HelixEducation> | Facebook<
> https://www.facebook.com/HelixEducation> | LinkedIn<
> http://www.linkedin.com/company/3609946>
>