You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by techbysample <tu...@netmille.com> on 2017/11/02 00:26:21 UTC
GA Grid: Request to contribute GA library to Apache Ignite
Igniters,
GA Grid (Beta) is an in memory Genetic Algorithm (GA) for Apache Ignite used
to solve complex problems by simulating biological evolution.
GA's are a form of Machine Learning (ML), excellent for finding an optimal
solution, among possibly thousands (or more) candidate solutions for a given
domain.
GA Grid was developed by NetMillennium Inc, Inc outside of Apache Ignite
platform as a proof of concept to determine feasibility in the GA space.
In GA Grid, all genetic operations: Fitness Calculation, Crossover, and
Mutation are modeled as a ComputeTask for distributive behavior. Also,
these ComputeTasks leverage Apache Ignite's Affinity Colocation to route
ComputeJobs to respective nodes where Chromosomes are stored in cache.
After it's initial release, Denis Magda inquired about the possibility of
donating GA Grid to Apache Ignite. Per discussions with Denis, he believed
GA Grid would be well suited as an extension to Apache Ignite's ML library.
Currently, NetMillennium, Inc. has agreed to begin the process of donating
GA Grid to Apache Ignite.
With it's latest release, GA Grid enhances knowledge discovery by providing
custom SQL functions to 'pivot' genetic optimization results. This enables
improved visualizations inside Apache Zeppelin.
To learn more about GA Grid please visit:
https://github.com/techbysample/gagrid
Check out my recent post on how GA Grid for Ignite integrates with Zeppelin:
https://www.linkedin.com/post/edit/apache-ignite-visualize-ga-grid-solutions-deep-turik-campbell
Please advise.
Best Regards,
Turik Campbell
NetMillennium, Inc.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Also please check the coding guideline -
https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines
Code in PR should fits to this guideline.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik, Denis
Today GA Grid was merged into Apache Ignite.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Yury,
I have removed pom.good.xml from ml module.
Regards
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik,
Also please remove pom.good.xml from ml module.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Yury,
I have reviewed and removed unnecessary pom files.
In addition, I have updated my branch from master.
Thanks.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik,
I've made a review, please check it on the github, there is a question about
pom files there. Also please update your branch from master.
At this point we almost ready for merge.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
>
> Based on Denis' request, I understand that Oleg/Yuri will handle merging
> 'pull-request' into main codebase.
Exactly, I'm expecting the same.
--
Denis
On Tue, Feb 27, 2018 at 1:22 PM, techbysample <tu...@netmille.com> wrote:
> Denis,
>
> Good to hear! I am glad that GA Grid will officially become apart of
> Apache
> Ignite.
>
> Based on Denis' request, I understand that Oleg/Yuri will handle merging
> 'pull-request' into main codebase.
>
> Let me know if there is any action for me to take at this time.
>
> Also, Denis, I will be reaching out to you very soon to discuss next steps.
>
> Best,
> Turik Campbell
> NetMillennium, Inc.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Good to hear! I am glad that GA Grid will officially become apart of Apache
Ignite.
Based on Denis' request, I understand that Oleg/Yuri will handle merging
'pull-request' into main codebase.
Let me know if there is any action for me to take at this time.
Also, Denis, I will be reaching out to you very soon to discuss next steps.
Best,
Turik Campbell
NetMillennium, Inc.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Turik, All,
Congratulations, the vote passed by lazy consensus. Feel free to merge the
pull-request into the main code base. Oleg, Yuri, should help with this.
Ping me once this happens, I would discuss documentation and promotion
related questions.
Please help to brag about this:
https://twitter.com/denismagda/status/968253956286644224
--
Denis
On Fri, Feb 23, 2018 at 2:56 PM, techbysample <tu...@netmille.com> wrote:
> Denis,
>
> Great! Thanks for the update.
>
> Best,
> Turik Campbell
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Great! Thanks for the update.
Best,
Turik Campbell
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Turik, All,
The IP clearance form is ready [1], and the final vote on general@incubator
is on the go [2].
Once the vote passes (72 hours), I'll finalize the process letting you know
when the code can be merged.
[1] http://incubator.apache.org/ip-clearance/ga-grid-ignite.html
[2]
http://apache-incubator-general.996316.n3.nabble.com/IP-CLEARANCE-Apache-Ignite-Genetic-Algorithms-Grid-td57791.html
--
Denis
On Thu, Feb 15, 2018 at 6:44 PM, techbysample <tu...@netmille.com> wrote:
> Denis,
>
> Thank you for providing status update. I look forward to hearing from you.
>
> Best,
> Turik Campbell
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Thank you for providing status update. I look forward to hearing from you.
Best,
Turik Campbell
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Guys,
Having troubles connecting to ASF subversion to fill in the IP clearance
form today. Something messy with a networking environment in my current
workplace.
Anyway, checked up that we have everything in place to complete the form
and finish the process. Please don't merge the pull-request until the IP
clearance vote passes.
I'll keep you posted.
--
Denis
On Wed, Feb 7, 2018 at 4:01 PM, techbysample <tu...@netmille.com> wrote:
> Denis,
>
> Thank's for following up.. Please let me know once the paperwork has been
> completed..
>
> In addition, I agree with releasing GA Grid as part of ML framework in
> Ignite 2.5.
>
> Just let me know if you require additional information beyond what I have
> provided.
>
> Regards,
> Turik Campbell
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Thank's for following up.. Please let me know once the paperwork has been
completed..
In addition, I agree with releasing GA Grid as part of ML framework in
Ignite 2.5.
Just let me know if you require additional information beyond what I have
provided.
Regards,
Turik Campbell
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by Denis Magda <dm...@apache.org>.
Sorry guys, the latest replies skimmed through my radar. I’ll arrange some time this or next week to do the paperwork. Let’s plan to release the GA library as a part of ML framework in Ignite 2.5. Luckily, we have a plenty of time for that.
—
Denis
> On Jan 31, 2018, at 5:00 AM, Yury Babak <y....@gmail.com> wrote:
>
> Denis,
>
> From my point of view PR looks good and ready for merge. So we waiting
> notification from you that all paperwork is done.
>
> Regards,
> Yury.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by Yury Babak <y....@gmail.com>.
Denis,
From my point of view PR looks good and ready for merge. So we waiting
notification from you that all paperwork is done.
Regards,
Yury.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Hello.
I sent the form previously on November 20, 2017 to secretary@apache.org and
cc'd you at emails: dmagda@gridgain.com, dmagda@apache.org.
Per the emails below, I understood the document was already received by
Craig L Russell.
Please advise if you still require additional information from me.
Thank you.
Best,
Turik Campbell
On 2017-11-20 17:22, Turik Campbell wrote:
> To whom it may concern,
>
> Attached you will find a copy of the 'Software Grant Agreement'
> to donate GA Grid to Apache Ignite.
>
> Please advise.
>
> Best,
> Turik Campbell
> NetMillennium, Inc.
> 866.300.2393
>
>
On 2017-11-20 18:37, Craig L Russell wrote:
> Dear Turik Campbell,
>
> This message acknowledges receipt of the following document, which has
> been filed in the Apache Software Foundation records:
>
> Software Grant from NetMillennium, Inc.
>
> -- Craig L Russell
> Secretary, Apache Software Foundation
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by Denis Magda <dm...@apache.org>.
Hi Turik,
The next step is to file a software grant and navigate through the IP clearance process:
http://incubator.apache.org/ip-clearance/ip-clearance-template.html <http://incubator.apache.org/ip-clearance/ip-clearance-template.html>
Once this is done, the GA Grid will be officially accepted and released in the next Ignite version.
I’ll execute the IP clearance process as a PMC representative while I would ask you to file a software grant to the ASF:
* Grant template: http://www.apache.org/licenses/software-grant-template.pdf
* Submission: http://www.apache.org/licenses/#submitting
—
Denis
> On Jan 25, 2018, at 3:01 PM, techbysample <tu...@netmille.com> wrote:
>
> Denis/Yury,
>
> Please advise on next steps.
>
> JIRA: https://issues.apache.org/jira/browse/IGNITE-6899
>
> Per recent comments from Oleg Ignatenko:
>
> "...most recent change looks good to me: the typo in MovieFitnessFunction is
> fixed and the rest of code is the same as it was ie everything now looks
> right.."
>
> Please advise.
>
> Best,
> Turik Campbell
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Next Steps: GA Grid: Request to contribute GA library to Apache
Ignite
Posted by techbysample <tu...@netmille.com>.
Denis/Yury,
Please advise on next steps.
JIRA: https://issues.apache.org/jira/browse/IGNITE-6899
Per recent comments from Oleg Ignatenko:
"...most recent change looks good to me: the typo in MovieFitnessFunction is
fixed and the rest of code is the same as it was ie everything now looks
right.."
Please advise.
Best,
Turik Campbell
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Igor,
Thanks! I am able to update page as expected now.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by Igor Sapego <is...@apache.org>.
It is in a drop-down list that appears when you click on your
name, not where you are looking for it.
Best Regards,
Igor
On Sat, Dec 2, 2017 at 1:02 AM, techbysample <tu...@netmille.com> wrote:
> Denis,
>
> Up on logging in, I do not see the Admin panel that you described.
>
> I have attached a screen shot for you review.
>
>
> <http://apache-ignite-developers.2346864.n4.nabble.
> com/file/t375/ReadMeQuestion.png>
>
> There is only 'Suggest Edits' option.
>
> Is there something I missed?
>
> Please advise.
>
> Best,
> Turik
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Up on logging in, I do not see the Admin panel that you described.
I have attached a screen shot for you review.
<http://apache-ignite-developers.2346864.n4.nabble.com/file/t375/ReadMeQuestion.png>
There is only 'Suggest Edits' option.
Is there something I missed?
Please advise.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Akmal, yes, the updates need to be checked:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document <https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document>
An actual reiview though can take place before or after a doc goes online.
Turik,
You can edit the page directly. Once you log in under your account, click on your name to unveil a drop-down menu and select “Admin panel” from there. Then locate the GA page in the docs tree and edit it.
—
Denis
> On Dec 1, 2017, at 6:22 AM, Akmal Chaudhri <ak...@gridgain.com> wrote:
>
> Updates need to be checked and approved before going live?
>
> On 1 December 2017 at 04:11, techbysample <tu...@netmille.com> wrote:
>
>> Denis,
>>
>> Thanks. In review, I noticed that when I edit page:
>>
>> "https://apacheignite.readme.io/v2.3/docs/genetic-algorithms", my updates
>> do
>> not appear instantly.
>>
>> Is this the correct behavior of ReadMe?
>>
>> I assumed that updates were 'instant' akin to blogging..
>>
>> Please advise.
>>
>> Regards,
>> Turik
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by Akmal Chaudhri <ak...@gridgain.com>.
Updates need to be checked and approved before going live?
On 1 December 2017 at 04:11, techbysample <tu...@netmille.com> wrote:
> Denis,
>
> Thanks. In review, I noticed that when I edit page:
>
> "https://apacheignite.readme.io/v2.3/docs/genetic-algorithms", my updates
> do
> not appear instantly.
>
> Is this the correct behavior of ReadMe?
>
> I assumed that updates were 'instant' akin to blogging..
>
> Please advise.
>
> Regards,
> Turik
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Thanks. In review, I noticed that when I edit page:
"https://apacheignite.readme.io/v2.3/docs/genetic-algorithms", my updates do
not appear instantly.
Is this the correct behavior of ReadMe?
I assumed that updates were 'instant' akin to blogging..
Please advise.
Regards,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Documentation for GA Grid: GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Hi Turik,
> If so, I thought a section called 'Genetic Algorithm (ie: GA Grid)' could
> go under the Machine Learning topic.
> https://apacheignite.readme.io/docs/machine-learning <https://apacheignite.readme.io/docs/machine-learning>
This is a right approach. I’ve granted you the access to readme and created a hidden page template:
https://apacheignite.readme.io/v2.3/docs/genetic-algorithms <https://apacheignite.readme.io/v2.3/docs/genetic-algorithms>
You’ll be able to see and edit the page after signing in.
—
Denis
> On Nov 29, 2017, at 2:01 PM, techbysample <tu...@netmille.com> wrote:
>
> Denis,
>
> Where do you want me to implement documenation for GA Grid?
> (https://issues.apache.org/jira/browse/IGNITE-6899)
> I have implemented examples, but believe more information is necessary..
>
> IE: I see that when Persistent Store was implemented docs were
> placed here:
> https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Overview
> But, not really sure.
>
> Does it make since at this stage in process to implement on Ignite's ReadMe
> pages?
> If so, I thought a section called 'Genetic Algorithm (ie: GA Grid)' could
> go under the Machine Learning topic.
> https://apacheignite.readme.io/docs/machine-learning
>
>
> Please advise, and if necessary grant appropriate access.
>
> Regards,
> Turik
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Documentation for GA Grid: GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Where do you want me to implement documenation for GA Grid?
(https://issues.apache.org/jira/browse/IGNITE-6899)
I have implemented examples, but believe more information is necessary..
IE: I see that when Persistent Store was implemented docs were
placed here:
https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Overview
But, not really sure.
Does it make since at this stage in process to implement on Ignite's ReadMe
pages?
If so, I thought a section called 'Genetic Algorithm (ie: GA Grid)' could
go under the Machine Learning topic.
https://apacheignite.readme.io/docs/machine-learning
Please advise, and if necessary grant appropriate access.
Regards,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik,
For making pull-request you should perform the following steps:
1) create JIRA account(in case you dont have one)
(https://issues.apache.org/jira)
1.1) write to dev-list and ask contributors permission.
2) assign the ticket which I`ve created for you
(https://issues.apache.org/jira/browse/IGNITE-6899)
3) clone Apache Ignite and create new branch from master, for example branch
name could be ignite-6889
4) add whole GA Grid to package org.apache.ignite.ml.genetic
5) add some tests for GA Grid
6) add some examples to example module(your current tests are good for just
move them to example module)
7) create pull-request from your branch to our master
After those steps we will perform code review.And after this we could merge
PR.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis/Yury,
Thank you both for your valuable feedback.
I will begin reviewing software grant form and submit accordingly.
Also, I have created account in JIRA and submitted request for contributor
permission to
dev list.
In addition, I will follow steps related to creating a 'pull request'
provide by Yury in post:
http://apache-ignite-developers.2346864.n4.nabble.com/template/NamlServlet.jtp?macro=print_post&node=24207
I will follow up if I have additional questions.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Turik,
Basically, this all is being done in parallel:
- submit the software grant form.
- prepare and submit the pull-request for code review.
- I’ll start filling in the IP clearance form after the steps above are completed. A PMC member has to do it.
- once the pull-request is reviewed and *before* it’s merged, we need to initiate the contribution acceptance vote [2].
- when the vote passes we merge the pull-request, complete the IP Clearance form and announce the new addition.
Let me know if you still have any questions.
[1] http://www.apache.org/licenses/#clas <http://www.apache.org/licenses/#clas>
[2] http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Accept-Contribution-of-Ignite-Persistent-Store-td17896.html
—
Denis
> On Nov 13, 2017, at 9:54 PM, techbysample <tu...@netmille.com> wrote:
>
> Denis,
>
> Ok. It's not clear what stage we are in this process.
>
> Do I need to fill out the IP Clearance form here?
> http://incubator.apache.org/ip-clearance/ip-clearance-template.html
> If so, I will simply model what was done for Ignite Persistence Store.
>
> Also, What about the software grant form? When will it be provided?
>
> Will I follow steps/guidelines here for pull request?:
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-
> (Section: 1. Create GitHub pull-request)
>
> Please advise on general order of steps in this whole process..
>
> Regards,
> Turik
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis,
Ok. It's not clear what stage we are in this process.
Do I need to fill out the IP Clearance form here?
http://incubator.apache.org/ip-clearance/ip-clearance-template.html
If so, I will simply model what was done for Ignite Persistence Store.
Also, What about the software grant form? When will it be provided?
Will I follow steps/guidelines here for pull request?:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-
(Section: 1. Create GitHub pull-request)
Please advise on general order of steps in this whole process..
Regards,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
In addition to that, we need to push GA grid through the IP clearance process:
http://incubator.apache.org/ip-clearance/ip-clearance-template.html
This is how an IP clearance form looked like and process happened when GridGain was donating Ignite persistence:
http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
I’ll help with the formalities.
—
Denis
> On Nov 13, 2017, at 10:03 AM, Yury Babak <y....@gmail.com> wrote:
>
> Turik,
>
> From my point of view our first step is add GA Grid as is into package
> org.apache.ignite.ml.genetic in ML module.
>
> It shouldn't be a problem, but before this we should check that GA Grid fits
> to our codestyle.
>
> So please prepare pull-request with GA Grid.
>
> Also if nobody object I will create ticket in our JIRA for this first step.
>
> And also we have few formal steps, I hope Denis could help with them.
>
> Regards,
> Yury
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik,
From my point of view our first step is add GA Grid as is into package
org.apache.ignite.ml.genetic in ML module.
It shouldn't be a problem, but before this we should check that GA Grid fits
to our codestyle.
So please prepare pull-request with GA Grid.
Also if nobody object I will create ticket in our JIRA for this first step.
And also we have few formal steps, I hope Denis could help with them.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Yury,
Thanks for feedback.
I reviewed the Trainer API at:
https://github.com/apache/ignite/blob/db7697b17cf6eb94754edb2b5e200655a3610dc1/modules/ml/src/main/java/org/apache/ignite/ml/Trainer.java
and also recommend approach that new "GA trainers" should be implemented
that will use GA Grid".
Denis/Yury,
Please advise on next steps based on most recent posts.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik,
1) Yes, it`s correct.
2.a) Model API is available, Trainer API in
PR(https://github.com/apache/ignite/pull/2936) which should be merged today
or tomorrow.
2.b) Yes, Trainer generates Model. Here is the Trainer interface:
public interface Trainer<M extends Model, T> {
public M train(T data);
}
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis/Yury,
I updated my previous post slightly,
For clarity, Please simply respond to this post and disregard previous.
Upon review of your previous comments, please respond to my feedback :
1. I believe GA Grid can be implemented in separate package within ML
library and operate independently of other algorithms for use cases where
/only /GA is required.
2. I am still not totally clear concerning Trainer and Model relationship
in the ML API.
a. Since org.apache.ignite.ml.Trainer and
org.apache.ignite.ml.Model
are not available, when is it planned to be available?
Please advise when available.
b. Based on Yury's comments:
"...For both concepts we have API: org.apache.ignite.ml.Model and
org.apache.ignite.ml.Trainer. So if we want to use genetic algoritm for
model trainig we should implement specific trainer for each ML algorithms
like lin regression, kmean, decision tree and others.
For example let`s take a look on lin regression. Currently we have OLS
(Ordinary Least Squares) multiple linear regression. For this regression we
will have OLSRegressionModel and at least two possible trainers: analytical
trainer (a solution of matrix equation, analytical solution) and gradient
descent (numerical solution). And also we could implement GA trainer which
will use GA Grid... "
Do org.apache.ignite.ml.Trainer generate org.apache.ignite.ml.Models?
This part is still not clear..
Please advise and clarify accordingly.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis/Yury,
Upon review of your previous comments, please respond to my feedback :
1. I believe GA Grid can be implemented in separate package within ML
library and operate independently of other algorithms for use cases where
/only /GA is required.
2. I am still not totally clear concerning Trainer and Model relationship
in the ML API.
a. Am I correct that org.apache.ignite.ml.Trainer and
org.apache.ignite.ml.Model
API is not available as it is under development? Please advise.
b. Based on Yury's comments:
"...For both concepts we have API: org.apache.ignite.ml.Model and
org.apache.ignite.ml.Trainer. So if we want to use genetic algoritm for
model trainig we should implement specific trainer for each ML algorithms
like lin regression, kmean, decision tree and others.
For example let`s take a look on lin regression. Currently we have OLS
(Ordinary Least Squares) multiple linear regression. For this regression we
will have OLSRegressionModel and at least two possible trainers: analytical
trainer (a solution of matrix equation, analytical solution) and gradient
descent (numerical solution). And also we could implement GA trainer which
will use GA Grid... "
Do org.apache.ignite.ml.Trainer generate org.apache.ignite.ml.Models?
Please advise and clarify accordingly.
Best,
Turik
is an algorithm that generates
training algorithm builds a mode
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Yury,
Please see inline
> On Nov 7, 2017, at 12:11 PM, Yury Babak <y....@gmail.com> wrote:
>
> Denis,
>
> Let me clarify.
>
> Firstly, here gene is a single model coefficient(neuron weight, etc),
> chromosomes - whole model representation.
>
Sounds good. Actually a chromosome can be see as a model.
> Secondly GA should be implementation of Trainer API for each ML algorithm
> such as regression, clusterization, NNs, etc.
>
It’s optional, right? Initially there should be a way to run standard operations over chromosomes specific to GA only. Those operations/algorithms are crossover, fitness score, mutations. Do this 3 operations fit trainer API? We’re putting aside extended support of regression, clusterization, etc.
> And last but not least genetic algorithm does not fits for to Model API, so
> it shouldn't`t implement it. Generally genetic algorithm dont produce any
> predictive models.
>
I’m a bit confused here. Before we say that a chromosome is a model in terms of ML and it’s all about providing concrete trainer implementations.
—
Denis
> Regards,
> Yury
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Denis,
Let me clarify.
Firstly, here gene is a single model coefficient(neuron weight, etc),
chromosomes - whole model representation.
Secondly GA should be implementation of Trainer API for each ML algorithm
such as regression, clusterization, NNs, etc.
And last but not least genetic algorithm does not fits for to Model API, so
it shouldn't`t implement it. Generally genetic algorithm dont produce any
predictive models.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Yuri, Turik,
Considering the concept the ML is built around, it should be straightforward to adopt the genetic algorithms to it.
To be more specific, Genes and Chromosomes which are the central building blocks of GA turn out to be basic ML Models. All the standard genetic operations/algorithms such as mutation, crossover and fitness calculation correspond to ML Trainer.
If my understanding is correct, then we should add GA as a package to ML lib and implement all the basic Model and Trainer interfaces.
Sounds reasonable?
—
Denis
> On Nov 7, 2017, at 6:07 AM, Yury Babak <y....@gmail.com> wrote:
>
> Turik,
>
> Basically we have two main concepts the model and the trainer. Each machine
> learning method generates some model which could predict some result based
> on learning dataset. This model is just a function and model training is
> minimization of loss function, difference between model predictions and
> actual values.
>
> The model trainer is a mechanism for minimization of loss function. Usually
> for this purpose uses gradient descent or its variations like SGD.
>
> And here we could use genetic algorithm for minimization of loss function.
> Usining genetic algorithm we could find optimal values for neuron weights in
> neural network, cluster centers, regression coefficients, etc.
>
> For both contepts we have API: org.apache.ignite.ml.Model and
> org.apache.ignite.ml.Trainer. So if we want to use genetic algoritm for
> model trainig we should implement specific trainer for each ML algorithms
> like lin regression, kmean, decision tree and others.
>
> For example let`s take a look on lin regression. Currently we have OLS
> (Ordinary Least Squares) multiple linear regression. For this regression we
> will have OLSRegressionModel and at least two possible trainers: analytical
> trainer (a solution of matrix equation, analytical solution) and gradient
> descent (numerical solution). And also we could implement GA trainer which
> will use GA Grid.
>
> NB: this API is currently under development and right now lin regression
> doesn't use model and trainer API, we will refactor this algorithm in
> nearest future.
>
> Regards,
> Yury
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Turik,
Basically we have two main concepts the model and the trainer. Each machine
learning method generates some model which could predict some result based
on learning dataset. This model is just a function and model training is
minimization of loss function, difference between model predictions and
actual values.
The model trainer is a mechanism for minimization of loss function. Usually
for this purpose uses gradient descent or its variations like SGD.
And here we could use genetic algorithm for minimization of loss function.
Usining genetic algorithm we could find optimal values for neuron weights in
neural network, cluster centers, regression coefficients, etc.
For both contepts we have API: org.apache.ignite.ml.Model and
org.apache.ignite.ml.Trainer. So if we want to use genetic algoritm for
model trainig we should implement specific trainer for each ML algorithms
like lin regression, kmean, decision tree and others.
For example let`s take a look on lin regression. Currently we have OLS
(Ordinary Least Squares) multiple linear regression. For this regression we
will have OLSRegressionModel and at least two possible trainers: analytical
trainer (a solution of matrix equation, analytical solution) and gradient
descent (numerical solution). And also we could implement GA trainer which
will use GA Grid.
NB: this API is currently under development and right now lin regression
doesn't use model and trainer API, we will refactor this algorithm in
nearest future.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Denis/Yury,
I am in favor of the second approach. Also, I envision GA Grid implemented
into a separate package within the ML module.
If possible, It seem's the initial priority would be merging GA Grid into
the ML module such that it operates independently. Next, we can
discuss/prioritize how best other ML algorithms could utilize GA Grid.(ie:
implement trainers based on GA Grid as you mentioned)
I am not very familiar with 'trainer' concepts in ML but don't see it as an
issue.
Please advise.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Yury,
There is no rush. So if the community agrees to accept GA as a part of ML then I would go for the second suggested approach.
Turik, what’d you think about the second approach?
—
Denis
> On Nov 3, 2017, at 8:54 AM, Yury Babak <y....@gmail.com> wrote:
>
> Hi all,
>
> Please let me put some comments about GA Grid. Actually I like it, but
> currently it doesn't fit to our API.
>
> I`m not sure that we could merge GA Grid as is into ML module, but we see
> two possibilities.
>
> First is add GA Grid as separate module like ML module.
>
> Second is adapt this genetic algorithm as trainer for ML model(like
> regressions, clusterers, neural nets).
>
> Also we could use both approaches: add GA Grid as separate module and
> implement trainers based on GA Grid.
>
> Regards,
> Yury
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Yury Babak <y....@gmail.com>.
Hi all,
Please let me put some comments about GA Grid. Actually I like it, but
currently it doesn't fit to our API.
I`m not sure that we could merge GA Grid as is into ML module, but we see
two possibilities.
First is add GA Grid as separate module like ML module.
Second is adapt this genetic algorithm as trainer for ML model(like
regressions, clusterers, neural nets).
Also we could use both approaches: add GA Grid as separate module and
implement trainers based on GA Grid.
Regards,
Yury
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Denis Magda <dm...@apache.org>.
Hi guys,
Yes, in my opinion genetic algorithms developed by Turik perfectly fit our ML component as a separate package. Look, *ML* is a building block on top of Ignite distributed storage and compute grid. Same is true about GA grid. So, why don’t we merge GA into ML?
Nikita, Yuri and the rest of ML folks please chime in.
—
Denis
> On Nov 2, 2017, at 5:02 PM, techbysample <tu...@netmille.com> wrote:
>
> Dmitriy,
>
> Hello. Unfortunately, I am not sure that I fully understand your comments:
>
> "..Is my understanding correct that GA Grid is a vertical component for
> genetic algorithms? So far Ignite has been a horizontal product without any
> vertical functionality. I personally would like to keep it this way.."
>
> Would you please clarify?
>
> I simply view GA Grid as a software component that implements a distributive
> Genetic Algorithm (GA).
> GA Grid relies on Apache Ignite's major features: advanced clustering,
> compute grid, data grid, etc.
>
> Here is a diagram of how GA Grid relates to other components within Ignite:
>
> <http://apache-ignite-developers.2346864.n4.nabble.com/file/t375/GAIgniteComps.png>
>
> Based my earlier discussion with Denis M., I assumed GA Grid could be added
> to the collection of ML algorithms within ML Grid, since GA's are a type of
> 'Machine Learning" algorithms.
>
> If it is determined that GA Grid would not fit into Apache Ignite
> architecturally, I would consider including
> it as separate Apache project.
>
> Denis, would you please add your feedback as well?
>
> Please advise.
>
> Best,
> Turik
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by techbysample <tu...@netmille.com>.
Dmitriy,
Hello. Unfortunately, I am not sure that I fully understand your comments:
"..Is my understanding correct that GA Grid is a vertical component for
genetic algorithms? So far Ignite has been a horizontal product without any
vertical functionality. I personally would like to keep it this way.."
Would you please clarify?
I simply view GA Grid as a software component that implements a distributive
Genetic Algorithm (GA).
GA Grid relies on Apache Ignite's major features: advanced clustering,
compute grid, data grid, etc.
Here is a diagram of how GA Grid relates to other components within Ignite:
<http://apache-ignite-developers.2346864.n4.nabble.com/file/t375/GAIgniteComps.png>
Based my earlier discussion with Denis M., I assumed GA Grid could be added
to the collection of ML algorithms within ML Grid, since GA's are a type of
'Machine Learning" algorithms.
If it is determined that GA Grid would not fit into Apache Ignite
architecturally, I would consider including
it as separate Apache project.
Denis, would you please add your feedback as well?
Please advise.
Best,
Turik
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: GA Grid: Request to contribute GA library to Apache Ignite
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Hi Turik,
Is my understanding correct that GA Grid is a vertical component for
genetic algorithms? So far Ignite has been a horizontal product without any
vertical functionality. I personally would like to keep it this way (other
community members should chime in here).
I personally think it is great that GA Grid gets a lot of mention on Apache
Ignite website and has been a valuable integration for the Ignite community.
If you would like to join Apache, have you considered a separate Apache
project? This may help you build a community around your work and benefit
from the established Apache processes during incubation.
D.
On Wed, Nov 1, 2017 at 5:26 PM, techbysample <tu...@netmille.com> wrote:
> Igniters,
>
> GA Grid (Beta) is an in memory Genetic Algorithm (GA) for Apache Ignite
> used
> to solve complex problems by simulating biological evolution.
> GA's are a form of Machine Learning (ML), excellent for finding an optimal
> solution, among possibly thousands (or more) candidate solutions for a
> given
> domain.
>
> GA Grid was developed by NetMillennium Inc, Inc outside of Apache Ignite
> platform as a proof of concept to determine feasibility in the GA space.
> In GA Grid, all genetic operations: Fitness Calculation, Crossover, and
> Mutation are modeled as a ComputeTask for distributive behavior. Also,
> these ComputeTasks leverage Apache Ignite's Affinity Colocation to route
> ComputeJobs to respective nodes where Chromosomes are stored in cache.
>
> After it's initial release, Denis Magda inquired about the possibility of
> donating GA Grid to Apache Ignite. Per discussions with Denis, he believed
> GA Grid would be well suited as an extension to Apache Ignite's ML library.
> Currently, NetMillennium, Inc. has agreed to begin the process of donating
> GA Grid to Apache Ignite.
>
> With it's latest release, GA Grid enhances knowledge discovery by providing
> custom SQL functions to 'pivot' genetic optimization results. This enables
> improved visualizations inside Apache Zeppelin.
>
> To learn more about GA Grid please visit:
>
> https://github.com/techbysample/gagrid
>
> Check out my recent post on how GA Grid for Ignite integrates with
> Zeppelin:
>
> https://www.linkedin.com/post/edit/apache-ignite-visualize-
> ga-grid-solutions-deep-turik-campbell
>
> Please advise.
>
> Best Regards,
> Turik Campbell
> NetMillennium, Inc.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>