[Math] Kohonen's SOFM

[Gilles <gi...@harfang.homelinux.org>]

Hi.

I've just attached the proposed implementation:
  https://issues.apache.org/jira/browse/MATH-923

It is part of a new package "o.a.c.m.ml.neuralnet" that is supposed
to contain tools for creating artificial neural networks; however
the code has been written based on the single SOFM use case.

Please review.


Regards,
Gilles


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org

Re: [Math] Kohonen's SOFM

[Gilles <gi...@harfang.homelinux.org>]

On Fri, 03 Jan 2014 21:17:43 +0100, Thomas Neidhart wrote:
> On 12/28/2013 06:35 PM, Gilles wrote:
>> Hi.
>>
>> I've just attached the proposed implementation:
>>  https://issues.apache.org/jira/browse/MATH-923
>>
>> It is part of a new package "o.a.c.m.ml.neuralnet" that is supposed
>> to contain tools for creating artificial neural networks; however
>> the code has been written based on the single SOFM use case.
>
> Please find below some comments based on your first patch:
>
> - SquareNeighbourhood
>
> to my understanding, this is only useful for the case of a 2d 
> network,

Not necessarily. The concept can be extended to higher dimensions.
[But I don't know how useful it would be to have a 3D SOFM.]

> thus it should be in the package twod.
>
> - MapUtils
>
> As the utility methods there are very specific for SOFMs I would 
> suggest
> to move this to the package sofm and maybe even rename it to 
> SOFMUtils.

They are not specific to SOFM.
There are other "map" types (which would be implemented in packages
similar to "o.a.c.m.ml.neuralnet.sofm"). But methods in "MapUtils"
could still be called with them.
Some are completely general (e.g. "findBest"), others assume a 2D
network; but at this point, I'm not sure it would be worth splitting it
into several, dimension-specific, classes.

Perhaps that the package layout will need additional layers, e.g.
   o.a.c.m.ml.neural.map.sofm
   o.a.c.m.ml.neural.layer.single
   o.a.c.m.ml.neural.layer.multi
etc.
[Cf. comments below.]

Then MapUtils would indeed be moved to "map".

>
> - NeuronString / NeuronSquareMesh2D
>
> Somehow I like the level of abstraction between the very generic
> "Network" and these classes which represent a very specific neuronal
> net, i.e. a string of neurons in the 1D case and a mesh of neurons in
> the 2D case. At least it takes some time to understand the structure 
> and
> relationship between them and I was wondering if they could extend a
> Network as this is their main purpose.

Do you mean inheritance, instead of composition?

The initial implementation actually used inheritance, and led to
more cumbersome code.
Also, I think that the single accessor to the underlying "Network"
instance makes it easier to locate where one needs to be careful
about thread-safety.

[Moreover, I recall that the majority here favours composition over
inheritance.]

>
> - package sofm.util
>
> just a minor thing, but I would move the function classes there to 
> the
> sofm package and make them package private. They are only used by the
> respective factories and I wonder if there is any other use for them.

There will be other uses if other types of map are to be implemented.
But it's true then that those utilities should not be defined inside
the "sofm" package. Unless there is a suggestion for another location,
I'd leave them separate so they are more "visible" than just an
implementation detail of the SOFM code.

>
> - NeighbourhoodSizeFunctionFactory
>
> has the same javadoc as the LearningFactorFunctionFactory.

A missed copy/paste. Corrected.

>
> - general
>
> I understand that the main purpose of the neuronal network was to 
> create
> the foundation for the SOFM, but what I was missing for the network 
> was
> the part of the activation function.
>
> Right now, the network can be trained using the UpdateAction, but 
> then
> one has to analyze the neurons itself, which is fine for the SOFM 
> case.
> For the more general case, such an activation function can be used to
> generate the output of the network based on a given input. Do you 
> have
> plans to further add something like this?

Not in the near future. This was the reason of the caveat mentioned in
the OP. I have not implemented another kind of neural net.
At this point, I think that it would introduce complexities of unknown
impact if we'd want to make it more general than necessary for the
current use-case.

The path to further development which I though of would be to try to 
use
the "Network" as it is currently, and implement the required 
functionality
in other packages e.g.
   o.a.c.m.ml.neuralnet.feedforward

Then, if something is general enough, it could be moved to the 
"Network"
class, or a base (or perhaps intermediate) class could be created.

It could also appear that the current "Network" contains functionality
that is too specific (i.e. only useful for networks such as the SOFM).
It could also be that we'll figure out that it's better not too mix a
SOFM-like network implementation with multiple-layer networks (?)...

Similarly for "Neuron".

In particular, there is no notion of "activation" for the SOFM-type
of networks. In fact, although presented as a neural net, it's also
a clustering method. Another reason I'd be careful not to try and
generalize and otherwise extend those classes too early, with
concepts that may prove too "distant" (implementation-wise).

>
> - userguide
>
> There are really nice opportunities for graphical demos of a SOFM
> together with the already existing unit tests for the travelling
> salesman problem. Do you already work on something like this as
> otherwise I would be interested in creating such examples.

I have one class that should go to the "src/userguide" part of the
repository. But it's very crude (no graphics).
You are very welcome to make it more attractive!

>
> The new code is really interesting and opens up a lot of other
> opportunities, thanks for your work on this. I will definitely look
> further into this topic. From my side, feel free to start committing
> your current status, we can surely discuss further issues afterwards.

Thanks for the review.


Regards,
Gilles


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org

Re: [Math] Kohonen's SOFM

[Thomas Neidhart <th...@gmail.com>]

On 12/28/2013 06:35 PM, Gilles wrote:
> Hi.
> 
> I've just attached the proposed implementation:
>  https://issues.apache.org/jira/browse/MATH-923
> 
> It is part of a new package "o.a.c.m.ml.neuralnet" that is supposed
> to contain tools for creating artificial neural networks; however
> the code has been written based on the single SOFM use case.

Please find below some comments based on your first patch:

- SquareNeighbourhood

to my understanding, this is only useful for the case of a 2d network,
thus it should be in the package twod.

- MapUtils

As the utility methods there are very specific for SOFMs I would suggest
to move this to the package sofm and maybe even rename it to SOFMUtils.

- NeuronString / NeuronSquareMesh2D

Somehow I like the level of abstraction between the very generic
"Network" and these classes which represent a very specific neuronal
net, i.e. a string of neurons in the 1D case and a mesh of neurons in
the 2D case. At least it takes some time to understand the structure and
relationship between them and I was wondering if they could extend a
Network as this is their main purpose.

- package sofm.util

just a minor thing, but I would move the function classes there to the
sofm package and make them package private. They are only used by the
respective factories and I wonder if there is any other use for them.

- NeighbourhoodSizeFunctionFactory

has the same javadoc as the LearningFactorFunctionFactory.

- general

I understand that the main purpose of the neuronal network was to create
the foundation for the SOFM, but what I was missing for the network was
the part of the activation function.

Right now, the network can be trained using the UpdateAction, but then
one has to analyze the neurons itself, which is fine for the SOFM case.
For the more general case, such an activation function can be used to
generate the output of the network based on a given input. Do you have
plans to further add something like this?

- userguide

There are really nice opportunities for graphical demos of a SOFM
together with the already existing unit tests for the travelling
salesman problem. Do you already work on something like this as
otherwise I would be interested in creating such examples.


The new code is really interesting and opens up a lot of other
opportunities, thanks for your work on this. I will definitely look
further into this topic. From my side, feel free to start committing
your current status, we can surely discuss further issues afterwards.

Thomas

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org

Re: [Math] Kohonen's SOFM

[Thomas Neidhart <th...@gmail.com>]

On 12/28/2013 06:35 PM, Gilles wrote:
> Hi.
> 
> I've just attached the proposed implementation:
>  https://issues.apache.org/jira/browse/MATH-923
> 
> It is part of a new package "o.a.c.m.ml.neuralnet" that is supposed
> to contain tools for creating artificial neural networks; however
> the code has been written based on the single SOFM use case.
> 
> Please review.

Looks interesting, but please give me some time to review.

Thomas

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org