You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@horn.apache.org by Baran Topal <ba...@barantopal.com> on 2016/09/01 11:17:55 UTC
Re: Use vector instead of Iterable in neuron API
Hi Edward and team;
I was checking the code and some trouble while building it from maven.
I tried with master and end up different dependency problems. I also
went for incubator-horn-d88a785. The core problem is that for some
reason the pom content at the end of the file is not recognized.
<artifactItems>
<artifactItem>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<outputDirectory>${project.basedir}/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<outputDirectory>${project.basedir}/lib</outputDirectory>
</artifactItem>
</artifactItems>
<excludeTransitive>true</excludeTransitive>
<fileMode>755</fileMode>
And possibly, due to this, the test case I am writing is failing with
java.lang.NoClassDefFoundError:
org/apache/commons/collections/map/UnmodifiableMap
1) Are you sure that the artifact item place is correct? (This is
happening in d88a785 and master)
2) I am using intellij 14.1.4 and using jdk 1.8.0_45. I am using
bundled maven 3 setup in intellij. An addition is that I am under
Windows and the file mode seems not really accurate for me?
3) A newbie question: Do I need a hadoop stack to be in place to test the code?
4) Should we have a session some time and maybe, we can record the
session with these basic questions and setups so the threshold for
newbies would be faster and the project will have more attention?
Br.
2016-08-26 11:43 GMT+02:00 Edward J. Yoon <ed...@apache.org>:
> Hi,
>
> Good point. I was aimed to parallelization of neuron objects using
> multi-threading. The original purpose of iterable msgs, is related
> with memory consumption issue (message serialization). BTW, thread
> within thread seems possible [1].
>
> And, In forward() method, we usually computes the weighted sum. This
> computation can be parallelized and thread safe.
>
> 1. http://stackoverflow.com/questions/7224670/threads-within-threads-in-java
>
>
> On Fri, Aug 26, 2016 at 6:01 PM, Baran Topal <ba...@barantopal.com> wrote:
>> Hi;
>>
>> First message, please be kind :)
>>
>> Is thread safety an important aspect in this context? Vectors are
>> threadsafe but the complexity (e.g. removal of an element) might not
>> be very desirable for performant applications. I will try to test and
>> inform you between two solutions with tests once I have time.
>>
>> Br.
>>
>> 2016-08-26 2:40 GMT+02:00 Edward J. Yoon <ed...@samsung.com>:
>>> Hi forks,
>>>
>>> Our current neuron API is designed like:
>>> https://github.com/apache/incubator-horn/blob/master/README.md#programming-m
>>> odel
>>>
>>> In forward() method, each neuron receives the pairs of the inputs x1, x2,
>>> ... xn from other neurons and weights w1, w2, ... wn like below:
>>>
>>> public void forward(Iterable<M> messages) throws IOException;
>>>
>>> Instead of this, I suggest that we use just vector like below:
>>>
>>> /**
>>> * @param input vector from other neurons
>>> * /
>>> public void forward(Vector input) throws IOException;
>>>
>>> And, the neuron provides a getWeightVector() method that returns weight
>>> vector associated with itself. I think this is more make sense than current
>>> version, and more easy to use GPU in the future.
>>>
>>> What do you think?
>>>
>>> Thanks.
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>>
>>>
>>>
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
RE: Use vector instead of Iterable in neuron API
Posted by "Edward J. Yoon" <ed...@samsung.com>.
Sorry, I never tried to build on Windows. You can build master with following
command: % mvn install
Hadoop stack is required only when setting the fully distributed mode on the
cluster. Local mode doesn't requires Hadoop.
--
Best Regards, Edward J. Yoon
-----Original Message-----
From: Baran Topal [mailto:barantopal@barantopal.com]
Sent: Thursday, September 01, 2016 8:18 PM
To: dev@horn.incubator.apache.org
Subject: Re: Use vector instead of Iterable in neuron API
Hi Edward and team;
I was checking the code and some trouble while building it from maven.
I tried with master and end up different dependency problems. I also
went for incubator-horn-d88a785. The core problem is that for some
reason the pom content at the end of the file is not recognized.
<artifactItems>
<artifactItem>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<outputDirectory>${project.basedir}/lib</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<outputDirectory>${project.basedir}/lib</outputDirectory>
</artifactItem>
</artifactItems>
<excludeTransitive>true</excludeTransitive>
<fileMode>755</fileMode>
And possibly, due to this, the test case I am writing is failing with
java.lang.NoClassDefFoundError:
org/apache/commons/collections/map/UnmodifiableMap
1) Are you sure that the artifact item place is correct? (This is
happening in d88a785 and master)
2) I am using intellij 14.1.4 and using jdk 1.8.0_45. I am using
bundled maven 3 setup in intellij. An addition is that I am under
Windows and the file mode seems not really accurate for me?
3) A newbie question: Do I need a hadoop stack to be in place to test the
code?
4) Should we have a session some time and maybe, we can record the
session with these basic questions and setups so the threshold for
newbies would be faster and the project will have more attention?
Br.
2016-08-26 11:43 GMT+02:00 Edward J. Yoon <ed...@apache.org>:
> Hi,
>
> Good point. I was aimed to parallelization of neuron objects using
> multi-threading. The original purpose of iterable msgs, is related
> with memory consumption issue (message serialization). BTW, thread
> within thread seems possible [1].
>
> And, In forward() method, we usually computes the weighted sum. This
> computation can be parallelized and thread safe.
>
> 1. http://stackoverflow.com/questions/7224670/threads-within-threads-in-java
>
>
> On Fri, Aug 26, 2016 at 6:01 PM, Baran Topal <ba...@barantopal.com>
> wrote:
>> Hi;
>>
>> First message, please be kind :)
>>
>> Is thread safety an important aspect in this context? Vectors are
>> threadsafe but the complexity (e.g. removal of an element) might not
>> be very desirable for performant applications. I will try to test and
>> inform you between two solutions with tests once I have time.
>>
>> Br.
>>
>> 2016-08-26 2:40 GMT+02:00 Edward J. Yoon <ed...@samsung.com>:
>>> Hi forks,
>>>
>>> Our current neuron API is designed like:
>>> https://github.com/apache/incubator-horn/blob/master/README.md#programming-m
>>> odel
>>>
>>> In forward() method, each neuron receives the pairs of the inputs x1, x2,
>>> ... xn from other neurons and weights w1, w2, ... wn like below:
>>>
>>> public void forward(Iterable<M> messages) throws IOException;
>>>
>>> Instead of this, I suggest that we use just vector like below:
>>>
>>> /**
>>> * @param input vector from other neurons
>>> * /
>>> public void forward(Vector input) throws IOException;
>>>
>>> And, the neuron provides a getWeightVector() method that returns weight
>>> vector associated with itself. I think this is more make sense than
>>> current
>>> version, and more easy to use GPU in the future.
>>>
>>> What do you think?
>>>
>>> Thanks.
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>>
>>>
>>>
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon