You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Andy Chambers <an...@fundingcircle.com> on 2016/02/10 07:22:27 UTC
Allow user-specified class-loader
Hey Folks,
I'm trying to build some tooling to make writing jobs in Clojure a little bit
more interactive. One feature that I think would help a lot is to allow
specification of a ClassLoader that is able to find classes defined dynamically
in Clojure.
Would you consider a feature/patch that adds a config parameter for this that
was respected everywhere Class/forName is called?
Is there anything else I should consider?
Thanks,
Andy
Andy Chambers | Full-Stack Software Developer
andy.chambers@fundingcircle.com | 707-205-6502 (m)
747 Front St, 4th Fl | San Francisco, CA 94111
Our Mission: T o build a better financial world
Unless specifically indicated, this e-mail is not an offer to sell or a
solicitation of any investment products or other financial product or service,
an official confirmation of any transaction, or an official statement of Funding
Circle USA. This e-mail is meant only for the intended recipient of this transmission, and
contains trade secret and strictly confidential information belonging to the
sender. It is unlawful for unauthorized individuals to review, use, copy,
disclose, or disseminate confidential information. If you have received this
e-mail in error, please notify the sender immediately by telephone at 857.285.1263 or by return email and promptly delete this message from your system.
Re: Allow user-specified class-loader
Posted by Rick Mangi <ri...@chartbeat.com>.
This is why we went the route of using clojure from the start and not calling out to clojure from a java task.
> On Feb 10, 2016, at 9:11 PM, Andy Chambers <an...@fundingcircle.com> wrote:
>
> Have you figured out a workflow where you can run jobs from rhe repl during
> development?
Re: Allow user-specified class-loader
Posted by Andy Chambers <an...@fundingcircle.com>.
Have you figured out a workflow where you can run jobs from rhe repl during
development?
Re: Allow user-specified class-loader
Posted by ri...@chartbeat.com.
Why aren't you just writing samza tasks directly in clojure using gen-class and implementing the samza interfaces? Works great for us. Much simpler.
> On Feb 10, 2016, at 7:39 PM, Jacob Maes <ja...@gmail.com> wrote:
>
> Hey Andy,
>
> That's good news! Do you mind sharing the revelation for any other Samza
> users in Clojure-land?
>
> -Jake
>
> On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers <
> andy.chambers@fundingcircle.com> wrote:
>
>> My co-worker has figured out how to do this without any changes required in
>> samza.
>> Andy Chambers | Full-Stack Software Developer
>>
>> andy.chambers@fundingcircle.com | 707-205-6502 (m)
>>
>>
>>
>>
>> 747 Front St, 4th Fl | San Francisco, CA 94111
>>
>> Our Mission: T o build a better financial world
>>
>>
>>
>>
>>
>>
>> Unless specifically indicated, this e-mail is not an offer to sell or a
>> solicitation of any investment products or other financial product or
>> service,
>> an official confirmation of any transaction, or an official statement of
>> Funding
>> Circle USA. This e-mail is meant only for the intended recipient of this
>> transmission, and
>> contains trade secret and strictly confidential information belonging to
>> the
>> sender. It is unlawful for unauthorized individuals to review, use, copy,
>> disclose, or disseminate confidential information. If you have received
>> this
>> e-mail in error, please notify the sender immediately by telephone at
>> 857.285.1263 or by return email and promptly delete this message from
>> your system.
>>
>>
>>
>> On 9 February 2016 at 23:47, Andy Chambers <
>> andy.chambers@fundingcircle.com > wrote:
>> No Problem. I'm new to samza so it is entirely possible there's already a
>> way to
>> do this that I'm missing.
>> What I'm aiming for, is to implement the features described in the README
>> of my
>> project https://github.com/ cddr/samza-config . For discussion purposes,
>> I'll copy the example here...
>> (ns example.word-counter
>> (:require
>> [samza-config.core :refer [stateful-task key-value-store]]))
>>
>> (defn count-words [store sentence output]
>> (doseq [word (split sentence)]
>> (update-in store word inc)
>> (output :word-count {:word word
>> :count (get store word)})))
>>
>> (defjob word-counter
>> {:inputs [(topic “words”)]
>> :outputs [(topic “word-counts”)]
>> :storage (key-value-store :word-counts)
>> :task (stateful-task
>> (fn [store input output]
>> (count-words store input output)))})
>>
>> I thought the easiest way to implement this would be for defjob to setup
>> some
>> metadata that points to implementations of StreamTask/InitableTask and
>> have a
>> custom ConfigFactory that can return the config for a named job.
>> Here is the (in development) implementation of ConfigFactory
>> (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let
>> [job (find-job uri)] (println “found job:” job) job)))
>> Unfortunately when I run the job runner, passing the name of this class as
>> the
>> --config-factory, I get the following stacktrace. I believe this is because
>> Clojure uses a special “DynamicClassLoader” to find these classes. That is
>> what
>> I'd set the classloader to if there was the option to do so.
>> $ lein do run -m samza-config.job example.jobs.hello-world Running samza
>> job: notifications.jobs.send-email Config Factory: samza_config.job.
>> JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader
>> 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in
>> thread “main” java.lang. ClassNotFoundException: samza_config.job.
>> JobConfigFactory, compiling:(/private/var/ folders/yt/
>> ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) at
>> clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler.
>> loadFile(Compiler.java:7317) at clojure.main$load_script.
>> invokeStatic(main.clj:275) at clojure.main$init_opt.
>> invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277)
>> at clojure.main$initialize. invokeStatic(main.clj:308) at
>> clojure.main$null_opt. invokeStatic(main.clj:342) at
>> clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main.
>> invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) at
>> clojure.lang.RestFn.invoke( RestFn.java:421) at
>> clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn.
>> applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at
>> clojure.main.main(main.java: 37) Caused by: java.lang.
>> ClassNotFoundException: samza_config.job. JobConfigFactory at
>> java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at
>> java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at
>> sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at
>> java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at
>> java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Class.
>> java:264) at org.apache.samza.util. CommandLine.loadConfig(
>> CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner.
>> scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at
>> samza_config.job$_main. invokeStatic(job.clj:102) at
>> samza_config.job$_main. doInvoke(job.clj:96) at clojure.lang.RestFn.invoke(
>> RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at
>> user$eval63.invokeStatic(form- init23293643677066296.clj:1) at
>> user$eval63.invoke(form- init23293643677066296.clj:1) at
>> clojure.lang.Compiler.eval( Compiler.java:6927) at
>> clojure.lang.Compiler.eval( Compiler.java:6917) at
>> clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more
>> That repo also has similarly defined implementations of serde factories
>> that
>> serialize/deserialize avro messages and check them against the confluent
>> schema
>> registry.
>> The biggest pain point IMO is the requirement to build a jar to run a job
>> during
>> development. I think it would be possible to define these classes in Java
>> and
>> have them call into a Clojure API but that's basically what I'm trying to
>> avoid
>> as I'm more likely to get the Java wrong.
>> Thanks for taking the time to understand this request.
>> Cheers, AndyAndy Chambers | Full-Stack Software Developer
>>
>> andy.chambers@fundingcircle. com | 707-205-6502 (m)
>>
>>
>>
>>
>> 747 Front St, 4th Fl | San Francisco, CA 94111
>>
>> Our Mission: T o build a better financial world
>>
>>
>>
>>
>>
>>
>> Unless specifically indicated, this e-mail is not an offer to sell or a
>> solicitation of any investment products or other financial product or
>> service,
>> an official confirmation of any transaction, or an official statement of
>> Funding
>> Circle USA. This e-mail is meant only for the intended recipient of this
>> transmission, and
>> contains trade secret and strictly confidential information belonging to
>> the
>> sender. It is unlawful for unauthorized individuals to review, use, copy,
>> disclose, or disseminate confidential information. If you have received
>> this
>> e-mail in error, please notify the sender immediately by telephone at
>> 857.285.1263 or by return email and promptly delete this message from
>> your system.
>>
>>
>> On 9 February 2016 at 22:46, Yi Pan < nickpan47@gmail.com > wrote:
>> Hi, Andy,
>>
>> Forgive me for my ignorance on the topic of Clojure. Could you give some
>> simple example that Config and Serde Factories are in "Clojure-land" and
>> how would a customized ClassLoader help in this case?
>>
>> Thanks a lot!
>>
>> -Yi
>>
>> On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
>> andy.chambers@fundingcircle. com > wrote:
>>
>>> That would help but if I understand it correctly, it would still be
>>> necessary to implement things like Config and Serde Factories in Java. It
>>> would be nice to stay in "Clojure-land" to implement those too.
>>>
>>> *Andy Chambers* | Full-Stack Software Developer
>>>
>>> * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com
>>> * |
>>> 707-205-6502 (m)
>>>
>>>
>>> 747 Front St, 4th Fl | San Francisco, CA 94111
>>>
>>> *Our Mission: **T**o build a better financial world*
>>>
>>>
>>> Unless specifically indicated, this e-mail is not an offer to sell or a
>>> solicitation of any investment products or other financial product or
>>> service, an official confirmation of any transaction, or an official
>>> statement of Funding Circle USA. This e-mail is meant only for the
>>> intended recipient of this transmission, and contains trade secret and
>>> strictly confidential information belonging to the sender. It is unlawful
>>> for unauthorized individuals to review, use, copy, disclose, or
>> disseminate
>>> confidential information. If you have received this e-mail in error,
>> please
>>> notify the sender immediately by telephone at 857.285.1263 or by return
>>> email and promptly delete this message from your system.
>>>
>>>> On 9 February 2016 at 22:25, Yi Pan < nickpan47@gmail.com > wrote:
>>>>
>>>> Hi, Andy,
>>>>
>>>> I think that you are looking for the feature in SAMZA-697. Or are you
>>>> looking for something even more specific?
>>>>
>>>> On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
>>>> andy.chambers@fundingcircle. com > wrote:
>>>>
>>>>> Hey Folks,
>>>>> I'm trying to build some tooling to make writing jobs in Clojure a
>>> little
>>>>> bit
>>>>> more interactive. One feature that I think would help a lot is to
>> allow
>>>>> specification of a ClassLoader that is able to find classes defined
>>>>> dynamically
>>>>> in Clojure.
>>>>> Would you consider a feature/patch that adds a config parameter for
>>> this
>>>>> that
>>>>> was respected everywhere Class/forName is called?
>>>>> Is there anything else I should consider?
>>>>> Thanks,
>>>>> Andy
>>>>> Andy Chambers | Full-Stack Software Developer
>>>>>
>>>>> andy.chambers@fundingcircle. com | 707-205-6502 (m)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 747 Front St, 4th Fl | San Francisco, CA 94111
>>>>>
>>>>> Our Mission: T o build a better financial world
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Unless specifically indicated, this e-mail is not an offer to sell
>> or a
>>>>> solicitation of any investment products or other financial product or
>>>>> service,
>>>>> an official confirmation of any transaction, or an official statement
>>> of
>>>>> Funding
>>>>> Circle USA. This e-mail is meant only for the intended recipient of
>>> this
>>>>> transmission, and
>>>>> contains trade secret and strictly confidential information belonging
>>> to
>>>>> the
>>>>> sender. It is unlawful for unauthorized individuals to review, use,
>>> copy,
>>>>> disclose, or disseminate confidential information. If you have
>> received
>>>>> this
>>>>> e-mail in error, please notify the sender immediately by telephone at
>>>>> 857.285.1263 or by return email and promptly delete this message
>> from
>>>>> your system.
>>
Re: Allow user-specified class-loader
Posted by Yi Pan <ni...@gmail.com>.
Hi, Andy,
Great! Looking forward to your uploads!
-Yi
On Wed, Feb 10, 2016 at 6:02 PM, Andy Chambers <
andy.chambers@fundingcircle.com> wrote:
> We will release some open source when it's ready but the basic idea is to
> define a single task that implements all the interfaces, and have that
> delegate to clojure handlers.
> On Feb 10, 2016 4:39 PM, "Jacob Maes" <ja...@gmail.com> wrote:
>
> > Hey Andy,
> >
> > That's good news! Do you mind sharing the revelation for any other Samza
> > users in Clojure-land?
> >
> > -Jake
> >
> > On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers <
> > andy.chambers@fundingcircle.com> wrote:
> >
> > > My co-worker has figured out how to do this without any changes
> required
> > in
> > > samza.
> > > Andy Chambers | Full-Stack Software Developer
> > >
> > > andy.chambers@fundingcircle.com | 707-205-6502 (m)
> > >
> > >
> > >
> > >
> > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > >
> > > Our Mission: T o build a better financial world
> > >
> > >
> > >
> > >
> > >
> > >
> > > Unless specifically indicated, this e-mail is not an offer to sell or a
> > > solicitation of any investment products or other financial product or
> > > service,
> > > an official confirmation of any transaction, or an official statement
> of
> > > Funding
> > > Circle USA. This e-mail is meant only for the intended recipient of
> this
> > > transmission, and
> > > contains trade secret and strictly confidential information belonging
> to
> > > the
> > > sender. It is unlawful for unauthorized individuals to review, use,
> copy,
> > > disclose, or disseminate confidential information. If you have received
> > > this
> > > e-mail in error, please notify the sender immediately by telephone at
> > > 857.285.1263 or by return email and promptly delete this message from
> > > your system.
> > >
> > >
> > >
> > > On 9 February 2016 at 23:47, Andy Chambers <
> > > andy.chambers@fundingcircle.com > wrote:
> > > No Problem. I'm new to samza so it is entirely possible there's
> already a
> > > way to
> > > do this that I'm missing.
> > > What I'm aiming for, is to implement the features described in the
> README
> > > of my
> > > project https://github.com/ cddr/samza-config . For discussion
> purposes,
> > > I'll copy the example here...
> > > (ns example.word-counter
> > > (:require
> > > [samza-config.core :refer [stateful-task key-value-store]]))
> > >
> > > (defn count-words [store sentence output]
> > > (doseq [word (split sentence)]
> > > (update-in store word inc)
> > > (output :word-count {:word word
> > > :count (get store word)})))
> > >
> > > (defjob word-counter
> > > {:inputs [(topic “words”)]
> > > :outputs [(topic “word-counts”)]
> > > :storage (key-value-store :word-counts)
> > > :task (stateful-task
> > > (fn [store input output]
> > > (count-words store input output)))})
> > >
> > > I thought the easiest way to implement this would be for defjob to
> setup
> > > some
> > > metadata that points to implementations of StreamTask/InitableTask and
> > > have a
> > > custom ConfigFactory that can return the config for a named job.
> > > Here is the (in development) implementation of ConfigFactory
> > > (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let
> > > [job (find-job uri)] (println “found job:” job) job)))
> > > Unfortunately when I run the job runner, passing the name of this class
> > as
> > > the
> > > --config-factory, I get the following stacktrace. I believe this is
> > because
> > > Clojure uses a special “DynamicClassLoader” to find these classes. That
> > is
> > > what
> > > I'd set the classloader to if there was the option to do so.
> > > $ lein do run -m samza-config.job example.jobs.hello-world Running
> samza
> > > job: notifications.jobs.send-email Config Factory: samza_config.job.
> > > JobConfigFactory Class Loader: #object[sun.misc.Launcher$
> AppClassLoader
> > > 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in
> > > thread “main” java.lang. ClassNotFoundException: samza_config.job.
> > > JobConfigFactory, compiling:(/private/var/ folders/yt/
> > > ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296.
> clj:1:123)
> > at
> > > clojure.lang.Compiler.load( Compiler.java:7391) at
> clojure.lang.Compiler.
> > > loadFile(Compiler.java:7317) at clojure.main$load_script.
> > > invokeStatic(main.clj:275) at clojure.main$init_opt.
> > > invokeStatic(main.clj:277) at clojure.main$init_opt.invoke(
> main.clj:277)
> > > at clojure.main$initialize. invokeStatic(main.clj:308) at
> > > clojure.main$null_opt. invokeStatic(main.clj:342) at
> > > clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main.
> > > invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384)
> > at
> > > clojure.lang.RestFn.invoke( RestFn.java:421) at
> > > clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn.
> > > applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700)
> at
> > > clojure.main.main(main.java: 37) Caused by: java.lang.
> > > ClassNotFoundException: samza_config.job. JobConfigFactory at
> > > java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at
> > > java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at
> > > sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at
> > > java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at
> > > java.lang.Class.forName0( Native Method) at
> > java.lang.Class.forName(Class.
> > > java:264) at org.apache.samza.util. CommandLine.loadConfig(
> > > CommandLine.scala:66) at org.apache.samza.job.
> JobRunner$.main(JobRunner.
> > > scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at
> > > samza_config.job$_main. invokeStatic(job.clj:102) at
> > > samza_config.job$_main. doInvoke(job.clj:96) at
> > clojure.lang.RestFn.invoke(
> > > RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at
> > > user$eval63.invokeStatic(form- init23293643677066296.clj:1) at
> > > user$eval63.invoke(form- init23293643677066296.clj:1) at
> > > clojure.lang.Compiler.eval( Compiler.java:6927) at
> > > clojure.lang.Compiler.eval( Compiler.java:6917) at
> > > clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more
> > > That repo also has similarly defined implementations of serde factories
> > > that
> > > serialize/deserialize avro messages and check them against the
> confluent
> > > schema
> > > registry.
> > > The biggest pain point IMO is the requirement to build a jar to run a
> job
> > > during
> > > development. I think it would be possible to define these classes in
> Java
> > > and
> > > have them call into a Clojure API but that's basically what I'm trying
> to
> > > avoid
> > > as I'm more likely to get the Java wrong.
> > > Thanks for taking the time to understand this request.
> > > Cheers, AndyAndy Chambers | Full-Stack Software Developer
> > >
> > > andy.chambers@fundingcircle. com | 707-205-6502 (m)
> > >
> > >
> > >
> > >
> > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > >
> > > Our Mission: T o build a better financial world
> > >
> > >
> > >
> > >
> > >
> > >
> > > Unless specifically indicated, this e-mail is not an offer to sell or a
> > > solicitation of any investment products or other financial product or
> > > service,
> > > an official confirmation of any transaction, or an official statement
> of
> > > Funding
> > > Circle USA. This e-mail is meant only for the intended recipient of
> this
> > > transmission, and
> > > contains trade secret and strictly confidential information belonging
> to
> > > the
> > > sender. It is unlawful for unauthorized individuals to review, use,
> copy,
> > > disclose, or disseminate confidential information. If you have received
> > > this
> > > e-mail in error, please notify the sender immediately by telephone at
> > > 857.285.1263 or by return email and promptly delete this message from
> > > your system.
> > >
> > >
> > > On 9 February 2016 at 22:46, Yi Pan < nickpan47@gmail.com > wrote:
> > > Hi, Andy,
> > >
> > > Forgive me for my ignorance on the topic of Clojure. Could you give
> some
> > > simple example that Config and Serde Factories are in "Clojure-land"
> and
> > > how would a customized ClassLoader help in this case?
> > >
> > > Thanks a lot!
> > >
> > > -Yi
> > >
> > > On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
> > > andy.chambers@fundingcircle. com > wrote:
> > >
> > > > That would help but if I understand it correctly, it would still be
> > > > necessary to implement things like Config and Serde Factories in
> Java.
> > It
> > > > would be nice to stay in "Clojure-land" to implement those too.
> > > >
> > > > *Andy Chambers* | Full-Stack Software Developer
> > > >
> > > > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle.
> com
> > > >* |
> > > > 707-205-6502 (m)
> > > >
> > > >
> > > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > > >
> > > > *Our Mission: **T**o build a better financial world*
> > > >
> > > >
> > > > Unless specifically indicated, this e-mail is not an offer to sell
> or a
> > > > solicitation of any investment products or other financial product or
> > > > service, an official confirmation of any transaction, or an official
> > > > statement of Funding Circle USA. This e-mail is meant only for the
> > > > intended recipient of this transmission, and contains trade secret
> and
> > > > strictly confidential information belonging to the sender. It is
> > unlawful
> > > > for unauthorized individuals to review, use, copy, disclose, or
> > > disseminate
> > > > confidential information. If you have received this e-mail in error,
> > > please
> > > > notify the sender immediately by telephone at 857.285.1263 or by
> > return
> > > > email and promptly delete this message from your system.
> > > >
> > > > On 9 February 2016 at 22:25, Yi Pan < nickpan47@gmail.com > wrote:
> > > >
> > > > > Hi, Andy,
> > > > >
> > > > > I think that you are looking for the feature in SAMZA-697. Or are
> you
> > > > > looking for something even more specific?
> > > > >
> > > > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> > > > > andy.chambers@fundingcircle. com > wrote:
> > > > >
> > > > > > Hey Folks,
> > > > > > I'm trying to build some tooling to make writing jobs in Clojure
> a
> > > > little
> > > > > > bit
> > > > > > more interactive. One feature that I think would help a lot is to
> > > allow
> > > > > > specification of a ClassLoader that is able to find classes
> defined
> > > > > > dynamically
> > > > > > in Clojure.
> > > > > > Would you consider a feature/patch that adds a config parameter
> for
> > > > this
> > > > > > that
> > > > > > was respected everywhere Class/forName is called?
> > > > > > Is there anything else I should consider?
> > > > > > Thanks,
> > > > > > Andy
> > > > > > Andy Chambers | Full-Stack Software Developer
> > > > > >
> > > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > > > > >
> > > > > > Our Mission: T o build a better financial world
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Unless specifically indicated, this e-mail is not an offer to
> sell
> > > or a
> > > > > > solicitation of any investment products or other financial
> product
> > or
> > > > > > service,
> > > > > > an official confirmation of any transaction, or an official
> > statement
> > > > of
> > > > > > Funding
> > > > > > Circle USA. This e-mail is meant only for the intended recipient
> of
> > > > this
> > > > > > transmission, and
> > > > > > contains trade secret and strictly confidential information
> > belonging
> > > > to
> > > > > > the
> > > > > > sender. It is unlawful for unauthorized individuals to review,
> use,
> > > > copy,
> > > > > > disclose, or disseminate confidential information. If you have
> > > received
> > > > > > this
> > > > > > e-mail in error, please notify the sender immediately by
> telephone
> > at
> > > > > > 857.285.1263 or by return email and promptly delete this message
> > > from
> > > > > > your system.
> > > > >
> > > >
> > >
> >
>
Re: Allow user-specified class-loader
Posted by Andy Chambers <an...@fundingcircle.com>.
We will release some open source when it's ready but the basic idea is to
define a single task that implements all the interfaces, and have that
delegate to clojure handlers.
On Feb 10, 2016 4:39 PM, "Jacob Maes" <ja...@gmail.com> wrote:
> Hey Andy,
>
> That's good news! Do you mind sharing the revelation for any other Samza
> users in Clojure-land?
>
> -Jake
>
> On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers <
> andy.chambers@fundingcircle.com> wrote:
>
> > My co-worker has figured out how to do this without any changes required
> in
> > samza.
> > Andy Chambers | Full-Stack Software Developer
> >
> > andy.chambers@fundingcircle.com | 707-205-6502 (m)
> >
> >
> >
> >
> > 747 Front St, 4th Fl | San Francisco, CA 94111
> >
> > Our Mission: T o build a better financial world
> >
> >
> >
> >
> >
> >
> > Unless specifically indicated, this e-mail is not an offer to sell or a
> > solicitation of any investment products or other financial product or
> > service,
> > an official confirmation of any transaction, or an official statement of
> > Funding
> > Circle USA. This e-mail is meant only for the intended recipient of this
> > transmission, and
> > contains trade secret and strictly confidential information belonging to
> > the
> > sender. It is unlawful for unauthorized individuals to review, use, copy,
> > disclose, or disseminate confidential information. If you have received
> > this
> > e-mail in error, please notify the sender immediately by telephone at
> > 857.285.1263 or by return email and promptly delete this message from
> > your system.
> >
> >
> >
> > On 9 February 2016 at 23:47, Andy Chambers <
> > andy.chambers@fundingcircle.com > wrote:
> > No Problem. I'm new to samza so it is entirely possible there's already a
> > way to
> > do this that I'm missing.
> > What I'm aiming for, is to implement the features described in the README
> > of my
> > project https://github.com/ cddr/samza-config . For discussion purposes,
> > I'll copy the example here...
> > (ns example.word-counter
> > (:require
> > [samza-config.core :refer [stateful-task key-value-store]]))
> >
> > (defn count-words [store sentence output]
> > (doseq [word (split sentence)]
> > (update-in store word inc)
> > (output :word-count {:word word
> > :count (get store word)})))
> >
> > (defjob word-counter
> > {:inputs [(topic “words”)]
> > :outputs [(topic “word-counts”)]
> > :storage (key-value-store :word-counts)
> > :task (stateful-task
> > (fn [store input output]
> > (count-words store input output)))})
> >
> > I thought the easiest way to implement this would be for defjob to setup
> > some
> > metadata that points to implementations of StreamTask/InitableTask and
> > have a
> > custom ConfigFactory that can return the config for a named job.
> > Here is the (in development) implementation of ConfigFactory
> > (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let
> > [job (find-job uri)] (println “found job:” job) job)))
> > Unfortunately when I run the job runner, passing the name of this class
> as
> > the
> > --config-factory, I get the following stacktrace. I believe this is
> because
> > Clojure uses a special “DynamicClassLoader” to find these classes. That
> is
> > what
> > I'd set the classloader to if there was the option to do so.
> > $ lein do run -m samza-config.job example.jobs.hello-world Running samza
> > job: notifications.jobs.send-email Config Factory: samza_config.job.
> > JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader
> > 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in
> > thread “main” java.lang. ClassNotFoundException: samza_config.job.
> > JobConfigFactory, compiling:(/private/var/ folders/yt/
> > ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123)
> at
> > clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler.
> > loadFile(Compiler.java:7317) at clojure.main$load_script.
> > invokeStatic(main.clj:275) at clojure.main$init_opt.
> > invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277)
> > at clojure.main$initialize. invokeStatic(main.clj:308) at
> > clojure.main$null_opt. invokeStatic(main.clj:342) at
> > clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main.
> > invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384)
> at
> > clojure.lang.RestFn.invoke( RestFn.java:421) at
> > clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn.
> > applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at
> > clojure.main.main(main.java: 37) Caused by: java.lang.
> > ClassNotFoundException: samza_config.job. JobConfigFactory at
> > java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at
> > java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at
> > sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at
> > java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at
> > java.lang.Class.forName0( Native Method) at
> java.lang.Class.forName(Class.
> > java:264) at org.apache.samza.util. CommandLine.loadConfig(
> > CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner.
> > scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at
> > samza_config.job$_main. invokeStatic(job.clj:102) at
> > samza_config.job$_main. doInvoke(job.clj:96) at
> clojure.lang.RestFn.invoke(
> > RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at
> > user$eval63.invokeStatic(form- init23293643677066296.clj:1) at
> > user$eval63.invoke(form- init23293643677066296.clj:1) at
> > clojure.lang.Compiler.eval( Compiler.java:6927) at
> > clojure.lang.Compiler.eval( Compiler.java:6917) at
> > clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more
> > That repo also has similarly defined implementations of serde factories
> > that
> > serialize/deserialize avro messages and check them against the confluent
> > schema
> > registry.
> > The biggest pain point IMO is the requirement to build a jar to run a job
> > during
> > development. I think it would be possible to define these classes in Java
> > and
> > have them call into a Clojure API but that's basically what I'm trying to
> > avoid
> > as I'm more likely to get the Java wrong.
> > Thanks for taking the time to understand this request.
> > Cheers, AndyAndy Chambers | Full-Stack Software Developer
> >
> > andy.chambers@fundingcircle. com | 707-205-6502 (m)
> >
> >
> >
> >
> > 747 Front St, 4th Fl | San Francisco, CA 94111
> >
> > Our Mission: T o build a better financial world
> >
> >
> >
> >
> >
> >
> > Unless specifically indicated, this e-mail is not an offer to sell or a
> > solicitation of any investment products or other financial product or
> > service,
> > an official confirmation of any transaction, or an official statement of
> > Funding
> > Circle USA. This e-mail is meant only for the intended recipient of this
> > transmission, and
> > contains trade secret and strictly confidential information belonging to
> > the
> > sender. It is unlawful for unauthorized individuals to review, use, copy,
> > disclose, or disseminate confidential information. If you have received
> > this
> > e-mail in error, please notify the sender immediately by telephone at
> > 857.285.1263 or by return email and promptly delete this message from
> > your system.
> >
> >
> > On 9 February 2016 at 22:46, Yi Pan < nickpan47@gmail.com > wrote:
> > Hi, Andy,
> >
> > Forgive me for my ignorance on the topic of Clojure. Could you give some
> > simple example that Config and Serde Factories are in "Clojure-land" and
> > how would a customized ClassLoader help in this case?
> >
> > Thanks a lot!
> >
> > -Yi
> >
> > On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
> > andy.chambers@fundingcircle. com > wrote:
> >
> > > That would help but if I understand it correctly, it would still be
> > > necessary to implement things like Config and Serde Factories in Java.
> It
> > > would be nice to stay in "Clojure-land" to implement those too.
> > >
> > > *Andy Chambers* | Full-Stack Software Developer
> > >
> > > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com
> > >* |
> > > 707-205-6502 (m)
> > >
> > >
> > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > >
> > > *Our Mission: **T**o build a better financial world*
> > >
> > >
> > > Unless specifically indicated, this e-mail is not an offer to sell or a
> > > solicitation of any investment products or other financial product or
> > > service, an official confirmation of any transaction, or an official
> > > statement of Funding Circle USA. This e-mail is meant only for the
> > > intended recipient of this transmission, and contains trade secret and
> > > strictly confidential information belonging to the sender. It is
> unlawful
> > > for unauthorized individuals to review, use, copy, disclose, or
> > disseminate
> > > confidential information. If you have received this e-mail in error,
> > please
> > > notify the sender immediately by telephone at 857.285.1263 or by
> return
> > > email and promptly delete this message from your system.
> > >
> > > On 9 February 2016 at 22:25, Yi Pan < nickpan47@gmail.com > wrote:
> > >
> > > > Hi, Andy,
> > > >
> > > > I think that you are looking for the feature in SAMZA-697. Or are you
> > > > looking for something even more specific?
> > > >
> > > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> > > > andy.chambers@fundingcircle. com > wrote:
> > > >
> > > > > Hey Folks,
> > > > > I'm trying to build some tooling to make writing jobs in Clojure a
> > > little
> > > > > bit
> > > > > more interactive. One feature that I think would help a lot is to
> > allow
> > > > > specification of a ClassLoader that is able to find classes defined
> > > > > dynamically
> > > > > in Clojure.
> > > > > Would you consider a feature/patch that adds a config parameter for
> > > this
> > > > > that
> > > > > was respected everywhere Class/forName is called?
> > > > > Is there anything else I should consider?
> > > > > Thanks,
> > > > > Andy
> > > > > Andy Chambers | Full-Stack Software Developer
> > > > >
> > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > > > >
> > > > > Our Mission: T o build a better financial world
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Unless specifically indicated, this e-mail is not an offer to sell
> > or a
> > > > > solicitation of any investment products or other financial product
> or
> > > > > service,
> > > > > an official confirmation of any transaction, or an official
> statement
> > > of
> > > > > Funding
> > > > > Circle USA. This e-mail is meant only for the intended recipient of
> > > this
> > > > > transmission, and
> > > > > contains trade secret and strictly confidential information
> belonging
> > > to
> > > > > the
> > > > > sender. It is unlawful for unauthorized individuals to review, use,
> > > copy,
> > > > > disclose, or disseminate confidential information. If you have
> > received
> > > > > this
> > > > > e-mail in error, please notify the sender immediately by telephone
> at
> > > > > 857.285.1263 or by return email and promptly delete this message
> > from
> > > > > your system.
> > > >
> > >
> >
>
Re: Allow user-specified class-loader
Posted by Jacob Maes <ja...@gmail.com>.
Hey Andy,
That's good news! Do you mind sharing the revelation for any other Samza
users in Clojure-land?
-Jake
On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers <
andy.chambers@fundingcircle.com> wrote:
> My co-worker has figured out how to do this without any changes required in
> samza.
> Andy Chambers | Full-Stack Software Developer
>
> andy.chambers@fundingcircle.com | 707-205-6502 (m)
>
>
>
>
> 747 Front St, 4th Fl | San Francisco, CA 94111
>
> Our Mission: T o build a better financial world
>
>
>
>
>
>
> Unless specifically indicated, this e-mail is not an offer to sell or a
> solicitation of any investment products or other financial product or
> service,
> an official confirmation of any transaction, or an official statement of
> Funding
> Circle USA. This e-mail is meant only for the intended recipient of this
> transmission, and
> contains trade secret and strictly confidential information belonging to
> the
> sender. It is unlawful for unauthorized individuals to review, use, copy,
> disclose, or disseminate confidential information. If you have received
> this
> e-mail in error, please notify the sender immediately by telephone at
> 857.285.1263 or by return email and promptly delete this message from
> your system.
>
>
>
> On 9 February 2016 at 23:47, Andy Chambers <
> andy.chambers@fundingcircle.com > wrote:
> No Problem. I'm new to samza so it is entirely possible there's already a
> way to
> do this that I'm missing.
> What I'm aiming for, is to implement the features described in the README
> of my
> project https://github.com/ cddr/samza-config . For discussion purposes,
> I'll copy the example here...
> (ns example.word-counter
> (:require
> [samza-config.core :refer [stateful-task key-value-store]]))
>
> (defn count-words [store sentence output]
> (doseq [word (split sentence)]
> (update-in store word inc)
> (output :word-count {:word word
> :count (get store word)})))
>
> (defjob word-counter
> {:inputs [(topic “words”)]
> :outputs [(topic “word-counts”)]
> :storage (key-value-store :word-counts)
> :task (stateful-task
> (fn [store input output]
> (count-words store input output)))})
>
> I thought the easiest way to implement this would be for defjob to setup
> some
> metadata that points to implementations of StreamTask/InitableTask and
> have a
> custom ConfigFactory that can return the config for a named job.
> Here is the (in development) implementation of ConfigFactory
> (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let
> [job (find-job uri)] (println “found job:” job) job)))
> Unfortunately when I run the job runner, passing the name of this class as
> the
> --config-factory, I get the following stacktrace. I believe this is because
> Clojure uses a special “DynamicClassLoader” to find these classes. That is
> what
> I'd set the classloader to if there was the option to do so.
> $ lein do run -m samza-config.job example.jobs.hello-world Running samza
> job: notifications.jobs.send-email Config Factory: samza_config.job.
> JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader
> 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in
> thread “main” java.lang. ClassNotFoundException: samza_config.job.
> JobConfigFactory, compiling:(/private/var/ folders/yt/
> ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) at
> clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler.
> loadFile(Compiler.java:7317) at clojure.main$load_script.
> invokeStatic(main.clj:275) at clojure.main$init_opt.
> invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277)
> at clojure.main$initialize. invokeStatic(main.clj:308) at
> clojure.main$null_opt. invokeStatic(main.clj:342) at
> clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main.
> invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) at
> clojure.lang.RestFn.invoke( RestFn.java:421) at
> clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn.
> applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at
> clojure.main.main(main.java: 37) Caused by: java.lang.
> ClassNotFoundException: samza_config.job. JobConfigFactory at
> java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at
> java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at
> sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at
> java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at
> java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Class.
> java:264) at org.apache.samza.util. CommandLine.loadConfig(
> CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner.
> scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at
> samza_config.job$_main. invokeStatic(job.clj:102) at
> samza_config.job$_main. doInvoke(job.clj:96) at clojure.lang.RestFn.invoke(
> RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at
> user$eval63.invokeStatic(form- init23293643677066296.clj:1) at
> user$eval63.invoke(form- init23293643677066296.clj:1) at
> clojure.lang.Compiler.eval( Compiler.java:6927) at
> clojure.lang.Compiler.eval( Compiler.java:6917) at
> clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more
> That repo also has similarly defined implementations of serde factories
> that
> serialize/deserialize avro messages and check them against the confluent
> schema
> registry.
> The biggest pain point IMO is the requirement to build a jar to run a job
> during
> development. I think it would be possible to define these classes in Java
> and
> have them call into a Clojure API but that's basically what I'm trying to
> avoid
> as I'm more likely to get the Java wrong.
> Thanks for taking the time to understand this request.
> Cheers, AndyAndy Chambers | Full-Stack Software Developer
>
> andy.chambers@fundingcircle. com | 707-205-6502 (m)
>
>
>
>
> 747 Front St, 4th Fl | San Francisco, CA 94111
>
> Our Mission: T o build a better financial world
>
>
>
>
>
>
> Unless specifically indicated, this e-mail is not an offer to sell or a
> solicitation of any investment products or other financial product or
> service,
> an official confirmation of any transaction, or an official statement of
> Funding
> Circle USA. This e-mail is meant only for the intended recipient of this
> transmission, and
> contains trade secret and strictly confidential information belonging to
> the
> sender. It is unlawful for unauthorized individuals to review, use, copy,
> disclose, or disseminate confidential information. If you have received
> this
> e-mail in error, please notify the sender immediately by telephone at
> 857.285.1263 or by return email and promptly delete this message from
> your system.
>
>
> On 9 February 2016 at 22:46, Yi Pan < nickpan47@gmail.com > wrote:
> Hi, Andy,
>
> Forgive me for my ignorance on the topic of Clojure. Could you give some
> simple example that Config and Serde Factories are in "Clojure-land" and
> how would a customized ClassLoader help in this case?
>
> Thanks a lot!
>
> -Yi
>
> On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
> andy.chambers@fundingcircle. com > wrote:
>
> > That would help but if I understand it correctly, it would still be
> > necessary to implement things like Config and Serde Factories in Java. It
> > would be nice to stay in "Clojure-land" to implement those too.
> >
> > *Andy Chambers* | Full-Stack Software Developer
> >
> > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com
> >* |
> > 707-205-6502 (m)
> >
> >
> > 747 Front St, 4th Fl | San Francisco, CA 94111
> >
> > *Our Mission: **T**o build a better financial world*
> >
> >
> > Unless specifically indicated, this e-mail is not an offer to sell or a
> > solicitation of any investment products or other financial product or
> > service, an official confirmation of any transaction, or an official
> > statement of Funding Circle USA. This e-mail is meant only for the
> > intended recipient of this transmission, and contains trade secret and
> > strictly confidential information belonging to the sender. It is unlawful
> > for unauthorized individuals to review, use, copy, disclose, or
> disseminate
> > confidential information. If you have received this e-mail in error,
> please
> > notify the sender immediately by telephone at 857.285.1263 or by return
> > email and promptly delete this message from your system.
> >
> > On 9 February 2016 at 22:25, Yi Pan < nickpan47@gmail.com > wrote:
> >
> > > Hi, Andy,
> > >
> > > I think that you are looking for the feature in SAMZA-697. Or are you
> > > looking for something even more specific?
> > >
> > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> > > andy.chambers@fundingcircle. com > wrote:
> > >
> > > > Hey Folks,
> > > > I'm trying to build some tooling to make writing jobs in Clojure a
> > little
> > > > bit
> > > > more interactive. One feature that I think would help a lot is to
> allow
> > > > specification of a ClassLoader that is able to find classes defined
> > > > dynamically
> > > > in Clojure.
> > > > Would you consider a feature/patch that adds a config parameter for
> > this
> > > > that
> > > > was respected everywhere Class/forName is called?
> > > > Is there anything else I should consider?
> > > > Thanks,
> > > > Andy
> > > > Andy Chambers | Full-Stack Software Developer
> > > >
> > > > andy.chambers@fundingcircle. com | 707-205-6502 (m)
> > > >
> > > >
> > > >
> > > >
> > > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > > >
> > > > Our Mission: T o build a better financial world
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Unless specifically indicated, this e-mail is not an offer to sell
> or a
> > > > solicitation of any investment products or other financial product or
> > > > service,
> > > > an official confirmation of any transaction, or an official statement
> > of
> > > > Funding
> > > > Circle USA. This e-mail is meant only for the intended recipient of
> > this
> > > > transmission, and
> > > > contains trade secret and strictly confidential information belonging
> > to
> > > > the
> > > > sender. It is unlawful for unauthorized individuals to review, use,
> > copy,
> > > > disclose, or disseminate confidential information. If you have
> received
> > > > this
> > > > e-mail in error, please notify the sender immediately by telephone at
> > > > 857.285.1263 or by return email and promptly delete this message
> from
> > > > your system.
> > >
> >
>
Re: Allow user-specified class-loader
Posted by Andy Chambers <an...@fundingcircle.com>.
My co-worker has figured out how to do this without any changes required in
samza.
Andy Chambers | Full-Stack Software Developer
andy.chambers@fundingcircle.com | 707-205-6502 (m)
747 Front St, 4th Fl | San Francisco, CA 94111
Our Mission: T o build a better financial world
Unless specifically indicated, this e-mail is not an offer to sell or a
solicitation of any investment products or other financial product or service,
an official confirmation of any transaction, or an official statement of Funding
Circle USA. This e-mail is meant only for the intended recipient of this transmission, and
contains trade secret and strictly confidential information belonging to the
sender. It is unlawful for unauthorized individuals to review, use, copy,
disclose, or disseminate confidential information. If you have received this
e-mail in error, please notify the sender immediately by telephone at 857.285.1263 or by return email and promptly delete this message from your system.
On 9 February 2016 at 23:47, Andy Chambers < andy.chambers@fundingcircle.com > wrote:
No Problem. I'm new to samza so it is entirely possible there's already a way to
do this that I'm missing.
What I'm aiming for, is to implement the features described in the README of my
project https://github.com/ cddr/samza-config . For discussion purposes, I'll copy the example here...
(ns example.word-counter
(:require
[samza-config.core :refer [stateful-task key-value-store]]))
(defn count-words [store sentence output]
(doseq [word (split sentence)]
(update-in store word inc)
(output :word-count {:word word
:count (get store word)})))
(defjob word-counter
{:inputs [(topic “words”)]
:outputs [(topic “word-counts”)]
:storage (key-value-store :word-counts)
:task (stateful-task
(fn [store input output]
(count-words store input output)))})
I thought the easiest way to implement this would be for defjob to setup some
metadata that points to implementations of StreamTask/InitableTask and have a
custom ConfigFactory that can return the config for a named job.
Here is the (in development) implementation of ConfigFactory
(defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let [job (find-job uri)] (println “found job:” job) job)))
Unfortunately when I run the job runner, passing the name of this class as the
--config-factory, I get the following stacktrace. I believe this is because
Clojure uses a special “DynamicClassLoader” to find these classes. That is what
I'd set the classloader to if there was the option to do so.
$ lein do run -m samza-config.job example.jobs.hello-world Running samza job: notifications.jobs.send-email Config Factory: samza_config.job. JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in thread “main” java.lang. ClassNotFoundException: samza_config.job. JobConfigFactory, compiling:(/private/var/ folders/yt/ ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) at clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler. loadFile(Compiler.java:7317) at clojure.main$load_script. invokeStatic(main.clj:275) at clojure.main$init_opt. invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277) at clojure.main$initialize. invokeStatic(main.clj:308) at clojure.main$null_opt. invokeStatic(main.clj:342) at clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main. invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) at clojure.lang.RestFn.invoke( RestFn.java:421) at clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn. applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at clojure.main.main(main.java: 37) Caused by: java.lang. ClassNotFoundException: samza_config.job. JobConfigFactory at java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Class. java:264) at org.apache.samza.util. CommandLine.loadConfig( CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner. scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at samza_config.job$_main. invokeStatic(job.clj:102) at samza_config.job$_main. doInvoke(job.clj:96) at clojure.lang.RestFn.invoke( RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at user$eval63.invokeStatic(form- init23293643677066296.clj:1) at user$eval63.invoke(form- init23293643677066296.clj:1) at clojure.lang.Compiler.eval( Compiler.java:6927) at clojure.lang.Compiler.eval( Compiler.java:6917) at clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more
That repo also has similarly defined implementations of serde factories that
serialize/deserialize avro messages and check them against the confluent schema
registry.
The biggest pain point IMO is the requirement to build a jar to run a job during
development. I think it would be possible to define these classes in Java and
have them call into a Clojure API but that's basically what I'm trying to avoid
as I'm more likely to get the Java wrong.
Thanks for taking the time to understand this request.
Cheers, AndyAndy Chambers | Full-Stack Software Developer
andy.chambers@fundingcircle. com | 707-205-6502 (m)
747 Front St, 4th Fl | San Francisco, CA 94111
Our Mission: T o build a better financial world
Unless specifically indicated, this e-mail is not an offer to sell or a
solicitation of any investment products or other financial product or service,
an official confirmation of any transaction, or an official statement of Funding
Circle USA. This e-mail is meant only for the intended recipient of this transmission, and
contains trade secret and strictly confidential information belonging to the
sender. It is unlawful for unauthorized individuals to review, use, copy,
disclose, or disseminate confidential information. If you have received this
e-mail in error, please notify the sender immediately by telephone at 857.285.1263 or by return email and promptly delete this message from your system.
On 9 February 2016 at 22:46, Yi Pan < nickpan47@gmail.com > wrote:
Hi, Andy,
Forgive me for my ignorance on the topic of Clojure. Could you give some
simple example that Config and Serde Factories are in "Clojure-land" and
how would a customized ClassLoader help in this case?
Thanks a lot!
-Yi
On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
andy.chambers@fundingcircle. com > wrote:
> That would help but if I understand it correctly, it would still be
> necessary to implement things like Config and Serde Factories in Java. It
> would be nice to stay in "Clojure-land" to implement those too.
>
> *Andy Chambers* | Full-Stack Software Developer
>
> * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com >* |
> 707-205-6502 (m)
>
>
> 747 Front St, 4th Fl | San Francisco, CA 94111
>
> *Our Mission: **T**o build a better financial world*
>
>
> Unless specifically indicated, this e-mail is not an offer to sell or a
> solicitation of any investment products or other financial product or
> service, an official confirmation of any transaction, or an official
> statement of Funding Circle USA. This e-mail is meant only for the
> intended recipient of this transmission, and contains trade secret and
> strictly confidential information belonging to the sender. It is unlawful
> for unauthorized individuals to review, use, copy, disclose, or disseminate
> confidential information. If you have received this e-mail in error, please
> notify the sender immediately by telephone at 857.285.1263 or by return
> email and promptly delete this message from your system.
>
> On 9 February 2016 at 22:25, Yi Pan < nickpan47@gmail.com > wrote:
>
> > Hi, Andy,
> >
> > I think that you are looking for the feature in SAMZA-697. Or are you
> > looking for something even more specific?
> >
> > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> > andy.chambers@fundingcircle. com > wrote:
> >
> > > Hey Folks,
> > > I'm trying to build some tooling to make writing jobs in Clojure a
> little
> > > bit
> > > more interactive. One feature that I think would help a lot is to allow
> > > specification of a ClassLoader that is able to find classes defined
> > > dynamically
> > > in Clojure.
> > > Would you consider a feature/patch that adds a config parameter for
> this
> > > that
> > > was respected everywhere Class/forName is called?
> > > Is there anything else I should consider?
> > > Thanks,
> > > Andy
> > > Andy Chambers | Full-Stack Software Developer
> > >
> > > andy.chambers@fundingcircle. com | 707-205-6502 (m)
> > >
> > >
> > >
> > >
> > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > >
> > > Our Mission: T o build a better financial world
> > >
> > >
> > >
> > >
> > >
> > >
> > > Unless specifically indicated, this e-mail is not an offer to sell or a
> > > solicitation of any investment products or other financial product or
> > > service,
> > > an official confirmation of any transaction, or an official statement
> of
> > > Funding
> > > Circle USA. This e-mail is meant only for the intended recipient of
> this
> > > transmission, and
> > > contains trade secret and strictly confidential information belonging
> to
> > > the
> > > sender. It is unlawful for unauthorized individuals to review, use,
> copy,
> > > disclose, or disseminate confidential information. If you have received
> > > this
> > > e-mail in error, please notify the sender immediately by telephone at
> > > 857.285.1263 or by return email and promptly delete this message from
> > > your system.
> >
>
Re: Allow user-specified class-loader
Posted by Andy Chambers <an...@fundingcircle.com>.
No Problem. I'm new to samza so it is entirely possible there's already a
way to do this that I'm missing.
What I'm aiming for, is to implement the features described in the README
of my project https://github.com/cddr/samza-config. For discussion
purposes, I'll copy the example here...
(ns example.word-counter
(:require
[samza-config.core :refer [stateful-task key-value-store]]))
(defn count-words [store sentence output]
(doseq [word (split sentence)]
(update-in store word inc)
(output :word-count {:word word
:count (get store word)})))
(defjob word-counter
{:inputs [(topic "words")]
:outputs [(topic "word-counts")]
:storage (key-value-store :word-counts)
:task (stateful-task
(fn [store input output]
(count-words store input output)))})
I thought the easiest way to implement this would be for defjob to setup
some metadata that points to implementations of StreamTask/InitableTask and
have a custom ConfigFactory that can return the config for a named job.
Here is the (in development) implementation of ConfigFactory
(defrecord JobConfigFactory []
ConfigFactory
(getConfig [this uri]
(let [job (find-job uri)]
(println "found job:" job)
job)))
Unfortunately when I run the job runner, passing the name of this class as
the --config-factory, I get the following stacktrace. I believe this is
because Clojure uses a special "DynamicClassLoader" to find these classes.
That is what I'd set the classloader to if there was the option to do so.
$ lein do run -m samza-config.job example.jobs.hello-world
Running samza job: notifications.jobs.send-email
Config Factory: samza_config.job.JobConfigFactory
Class Loader: #object[sun.misc.Launcher$AppClassLoader 0x4aa298b7
sun.misc.Launcher$AppClassLoader@4aa298b7]
Exception in thread "main" java.lang.ClassNotFoundException:
samza_config.job.JobConfigFactory,
compiling:(/private/var/folders/yt/ch58t4q565g3vjvyj5wjfkt00000gp/T/form-init23293643677066296.clj:1:123)
at clojure.lang.Compiler.load(Compiler.java:7391)
at clojure.lang.Compiler.loadFile(Compiler.java:7317)
at clojure.main$load_script.invokeStatic(main.clj:275)
at clojure.main$init_opt.invokeStatic(main.clj:277)
at clojure.main$init_opt.invoke(main.clj:277)
at clojure.main$initialize.invokeStatic(main.clj:308)
at clojure.main$null_opt.invokeStatic(main.clj:342)
at clojure.main$null_opt.invoke(main.clj:339)
at clojure.main$main.invokeStatic(main.clj:421)
at clojure.main$main.doInvoke(main.clj:384)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.ClassNotFoundException:
samza_config.job.JobConfigFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.samza.util.CommandLine.loadConfig(CommandLine.scala:66)
at org.apache.samza.job.JobRunner$.main(JobRunner.scala:65)
at org.apache.samza.job.JobRunner.main(JobRunner.scala)
at samza_config.job$_main.invokeStatic(job.clj:102)
at samza_config.job$_main.doInvoke(job.clj:96)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:379)
at user$eval63.invokeStatic(form-init23293643677066296.clj:1)
at user$eval63.invoke(form-init23293643677066296.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6927)
at clojure.lang.Compiler.eval(Compiler.java:6917)
at clojure.lang.Compiler.load(Compiler.java:7379)
... 14 more
That repo also has similarly defined implementations of serde factories
that serialize/deserialize avro messages and check them against the
confluent schema registry.
The biggest pain point IMO is the requirement to build a jar to run a job
during development. I think it would be possible to define these classes in
Java and have them call into a Clojure API but that's basically what I'm
trying to avoid as I'm more likely to get the Java wrong.
Thanks for taking the time to understand this request.
Cheers,
Andy
*Andy Chambers* | Full-Stack Software Developer
*andy.chambers@fundingcircle.com <an...@fundingcircle.com>* |
707-205-6502 (m)
747 Front St, 4th Fl | San Francisco, CA 94111
*Our Mission: **T**o build a better financial world*
Unless specifically indicated, this e-mail is not an offer to sell or a
solicitation of any investment products or other financial product or
service, an official confirmation of any transaction, or an official
statement of Funding Circle USA. This e-mail is meant only for the
intended recipient of this transmission, and contains trade secret and
strictly confidential information belonging to the sender. It is unlawful
for unauthorized individuals to review, use, copy, disclose, or disseminate
confidential information. If you have received this e-mail in error, please
notify the sender immediately by telephone at 857.285.1263 or by return
email and promptly delete this message from your system.
On 9 February 2016 at 22:46, Yi Pan <ni...@gmail.com> wrote:
> Hi, Andy,
>
> Forgive me for my ignorance on the topic of Clojure. Could you give some
> simple example that Config and Serde Factories are in "Clojure-land" and
> how would a customized ClassLoader help in this case?
>
> Thanks a lot!
>
> -Yi
>
> On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
> andy.chambers@fundingcircle.com> wrote:
>
> > That would help but if I understand it correctly, it would still be
> > necessary to implement things like Config and Serde Factories in Java. It
> > would be nice to stay in "Clojure-land" to implement those too.
> >
> > *Andy Chambers* | Full-Stack Software Developer
> >
> > *andy.chambers@fundingcircle.com <an...@fundingcircle.com>* |
> > 707-205-6502 (m)
> >
> >
> > 747 Front St, 4th Fl | San Francisco, CA 94111
> >
> > *Our Mission: **T**o build a better financial world*
> >
> >
> > Unless specifically indicated, this e-mail is not an offer to sell or a
> > solicitation of any investment products or other financial product or
> > service, an official confirmation of any transaction, or an official
> > statement of Funding Circle USA. This e-mail is meant only for the
> > intended recipient of this transmission, and contains trade secret and
> > strictly confidential information belonging to the sender. It is unlawful
> > for unauthorized individuals to review, use, copy, disclose, or
> disseminate
> > confidential information. If you have received this e-mail in error,
> please
> > notify the sender immediately by telephone at 857.285.1263 or by return
> > email and promptly delete this message from your system.
> >
> > On 9 February 2016 at 22:25, Yi Pan <ni...@gmail.com> wrote:
> >
> > > Hi, Andy,
> > >
> > > I think that you are looking for the feature in SAMZA-697. Or are you
> > > looking for something even more specific?
> > >
> > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> > > andy.chambers@fundingcircle.com> wrote:
> > >
> > > > Hey Folks,
> > > > I'm trying to build some tooling to make writing jobs in Clojure a
> > little
> > > > bit
> > > > more interactive. One feature that I think would help a lot is to
> allow
> > > > specification of a ClassLoader that is able to find classes defined
> > > > dynamically
> > > > in Clojure.
> > > > Would you consider a feature/patch that adds a config parameter for
> > this
> > > > that
> > > > was respected everywhere Class/forName is called?
> > > > Is there anything else I should consider?
> > > > Thanks,
> > > > Andy
> > > > Andy Chambers | Full-Stack Software Developer
> > > >
> > > > andy.chambers@fundingcircle.com | 707-205-6502 (m)
> > > >
> > > >
> > > >
> > > >
> > > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > > >
> > > > Our Mission: T o build a better financial world
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Unless specifically indicated, this e-mail is not an offer to sell
> or a
> > > > solicitation of any investment products or other financial product or
> > > > service,
> > > > an official confirmation of any transaction, or an official statement
> > of
> > > > Funding
> > > > Circle USA. This e-mail is meant only for the intended recipient of
> > this
> > > > transmission, and
> > > > contains trade secret and strictly confidential information belonging
> > to
> > > > the
> > > > sender. It is unlawful for unauthorized individuals to review, use,
> > copy,
> > > > disclose, or disseminate confidential information. If you have
> received
> > > > this
> > > > e-mail in error, please notify the sender immediately by telephone at
> > > > 857.285.1263 or by return email and promptly delete this message
> from
> > > > your system.
> > >
> >
>
Re: Allow user-specified class-loader
Posted by Yi Pan <ni...@gmail.com>.
Hi, Andy,
Forgive me for my ignorance on the topic of Clojure. Could you give some
simple example that Config and Serde Factories are in "Clojure-land" and
how would a customized ClassLoader help in this case?
Thanks a lot!
-Yi
On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers <
andy.chambers@fundingcircle.com> wrote:
> That would help but if I understand it correctly, it would still be
> necessary to implement things like Config and Serde Factories in Java. It
> would be nice to stay in "Clojure-land" to implement those too.
>
> *Andy Chambers* | Full-Stack Software Developer
>
> *andy.chambers@fundingcircle.com <an...@fundingcircle.com>* |
> 707-205-6502 (m)
>
>
> 747 Front St, 4th Fl | San Francisco, CA 94111
>
> *Our Mission: **T**o build a better financial world*
>
>
> Unless specifically indicated, this e-mail is not an offer to sell or a
> solicitation of any investment products or other financial product or
> service, an official confirmation of any transaction, or an official
> statement of Funding Circle USA. This e-mail is meant only for the
> intended recipient of this transmission, and contains trade secret and
> strictly confidential information belonging to the sender. It is unlawful
> for unauthorized individuals to review, use, copy, disclose, or disseminate
> confidential information. If you have received this e-mail in error, please
> notify the sender immediately by telephone at 857.285.1263 or by return
> email and promptly delete this message from your system.
>
> On 9 February 2016 at 22:25, Yi Pan <ni...@gmail.com> wrote:
>
> > Hi, Andy,
> >
> > I think that you are looking for the feature in SAMZA-697. Or are you
> > looking for something even more specific?
> >
> > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> > andy.chambers@fundingcircle.com> wrote:
> >
> > > Hey Folks,
> > > I'm trying to build some tooling to make writing jobs in Clojure a
> little
> > > bit
> > > more interactive. One feature that I think would help a lot is to allow
> > > specification of a ClassLoader that is able to find classes defined
> > > dynamically
> > > in Clojure.
> > > Would you consider a feature/patch that adds a config parameter for
> this
> > > that
> > > was respected everywhere Class/forName is called?
> > > Is there anything else I should consider?
> > > Thanks,
> > > Andy
> > > Andy Chambers | Full-Stack Software Developer
> > >
> > > andy.chambers@fundingcircle.com | 707-205-6502 (m)
> > >
> > >
> > >
> > >
> > > 747 Front St, 4th Fl | San Francisco, CA 94111
> > >
> > > Our Mission: T o build a better financial world
> > >
> > >
> > >
> > >
> > >
> > >
> > > Unless specifically indicated, this e-mail is not an offer to sell or a
> > > solicitation of any investment products or other financial product or
> > > service,
> > > an official confirmation of any transaction, or an official statement
> of
> > > Funding
> > > Circle USA. This e-mail is meant only for the intended recipient of
> this
> > > transmission, and
> > > contains trade secret and strictly confidential information belonging
> to
> > > the
> > > sender. It is unlawful for unauthorized individuals to review, use,
> copy,
> > > disclose, or disseminate confidential information. If you have received
> > > this
> > > e-mail in error, please notify the sender immediately by telephone at
> > > 857.285.1263 or by return email and promptly delete this message from
> > > your system.
> >
>
Re: Allow user-specified class-loader
Posted by Andy Chambers <an...@fundingcircle.com>.
That would help but if I understand it correctly, it would still be
necessary to implement things like Config and Serde Factories in Java. It
would be nice to stay in "Clojure-land" to implement those too.
*Andy Chambers* | Full-Stack Software Developer
*andy.chambers@fundingcircle.com <an...@fundingcircle.com>* |
707-205-6502 (m)
747 Front St, 4th Fl | San Francisco, CA 94111
*Our Mission: **T**o build a better financial world*
Unless specifically indicated, this e-mail is not an offer to sell or a
solicitation of any investment products or other financial product or
service, an official confirmation of any transaction, or an official
statement of Funding Circle USA. This e-mail is meant only for the
intended recipient of this transmission, and contains trade secret and
strictly confidential information belonging to the sender. It is unlawful
for unauthorized individuals to review, use, copy, disclose, or disseminate
confidential information. If you have received this e-mail in error, please
notify the sender immediately by telephone at 857.285.1263 or by return
email and promptly delete this message from your system.
On 9 February 2016 at 22:25, Yi Pan <ni...@gmail.com> wrote:
> Hi, Andy,
>
> I think that you are looking for the feature in SAMZA-697. Or are you
> looking for something even more specific?
>
> On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
> andy.chambers@fundingcircle.com> wrote:
>
> > Hey Folks,
> > I'm trying to build some tooling to make writing jobs in Clojure a little
> > bit
> > more interactive. One feature that I think would help a lot is to allow
> > specification of a ClassLoader that is able to find classes defined
> > dynamically
> > in Clojure.
> > Would you consider a feature/patch that adds a config parameter for this
> > that
> > was respected everywhere Class/forName is called?
> > Is there anything else I should consider?
> > Thanks,
> > Andy
> > Andy Chambers | Full-Stack Software Developer
> >
> > andy.chambers@fundingcircle.com | 707-205-6502 (m)
> >
> >
> >
> >
> > 747 Front St, 4th Fl | San Francisco, CA 94111
> >
> > Our Mission: T o build a better financial world
> >
> >
> >
> >
> >
> >
> > Unless specifically indicated, this e-mail is not an offer to sell or a
> > solicitation of any investment products or other financial product or
> > service,
> > an official confirmation of any transaction, or an official statement of
> > Funding
> > Circle USA. This e-mail is meant only for the intended recipient of this
> > transmission, and
> > contains trade secret and strictly confidential information belonging to
> > the
> > sender. It is unlawful for unauthorized individuals to review, use, copy,
> > disclose, or disseminate confidential information. If you have received
> > this
> > e-mail in error, please notify the sender immediately by telephone at
> > 857.285.1263 or by return email and promptly delete this message from
> > your system.
>
Re: Allow user-specified class-loader
Posted by Yi Pan <ni...@gmail.com>.
Hi, Andy,
I think that you are looking for the feature in SAMZA-697. Or are you
looking for something even more specific?
On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers <
andy.chambers@fundingcircle.com> wrote:
> Hey Folks,
> I'm trying to build some tooling to make writing jobs in Clojure a little
> bit
> more interactive. One feature that I think would help a lot is to allow
> specification of a ClassLoader that is able to find classes defined
> dynamically
> in Clojure.
> Would you consider a feature/patch that adds a config parameter for this
> that
> was respected everywhere Class/forName is called?
> Is there anything else I should consider?
> Thanks,
> Andy
> Andy Chambers | Full-Stack Software Developer
>
> andy.chambers@fundingcircle.com | 707-205-6502 (m)
>
>
>
>
> 747 Front St, 4th Fl | San Francisco, CA 94111
>
> Our Mission: T o build a better financial world
>
>
>
>
>
>
> Unless specifically indicated, this e-mail is not an offer to sell or a
> solicitation of any investment products or other financial product or
> service,
> an official confirmation of any transaction, or an official statement of
> Funding
> Circle USA. This e-mail is meant only for the intended recipient of this
> transmission, and
> contains trade secret and strictly confidential information belonging to
> the
> sender. It is unlawful for unauthorized individuals to review, use, copy,
> disclose, or disseminate confidential information. If you have received
> this
> e-mail in error, please notify the sender immediately by telephone at
> 857.285.1263 or by return email and promptly delete this message from
> your system.