You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hama.apache.org by "Edward J. Yoon" <ed...@samsung.com> on 2016/02/03 00:49:45 UTC
RE: RE: RE: RE: RE: Do Hama support member member variable?
Hi, 步?云
Please try to use TRUNK version. and you can refer the
https://github.com/apache/hama/blob/master/examples/src/test/java/org/apache/hama/examples/CustomVertexReadWriteStateTest.java
and
https://github.com/apache/hama/blob/master/ml/src/main/java/org/apache/hama/ml/kcore/KCoreVertex.java.
Both code contains custom read/writeState use case.
--
Best Regards, Edward J. Yoon
-----Original Message-----
From: Edward J. Yoon [mailto:edwardyoon@apache.org]
Sent: Thursday, January 28, 2016 8:33 PM
To: user@hama.apache.org
Subject: Re: RE: RE: RE: RE: Do Hama support member member variable?
I filed on https://issues.apache.org/jira/browse/HAMA-982
Once it's fixed, I'll let you know.
On Thu, Jan 28, 2016 at 8:11 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Wow, you find the bug. Thanks ;)
>
>> at
>> org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>
> When the framework assign vertices to the proper machine at initial
> phase, vertex objects are transferred in serialized form. At this step
> user defined code won't work correctly. I'll fix soon.
>
> Anyway, you should able to manage an array of TextPair objects like below:
>
> private TextPair[] test = new TextPair[1];
>
> public void readState(DataInput in) throws IOException {
> int size = in.readInt();
> test = new TextPair[size];
> for(int i = 0; i < size; i++) {
> test[i].readFields(in);
> }
> }
>
> public void writeState(DataOutput out) throws IOException {
> out.writeInt(test.length);
> for(int i = 0; i < test.length; i++) {
> test[i].write(out);
> }
> }
>
> Thanks.
>
>
> On Thu, Jan 28, 2016 at 6:45 PM, 步?云 <ma...@qq.com> wrote:
>> Thanks for your reply. You do help me a lot.
>> I have tried to use two methods. Some problem are still bothering me.
>> When I use the first method of using Hadoop built-in writable classes, I
>> get a NullPointerException meaning the parents is null， even though I have
>> initialize parents. The code is like this.
>> static ArrayWritable parents= new ArrayWritable(TextPair.class);
>>
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeBoolean(match);
>> parents.write(out);
>> }
>>
>>
>> public void readState(DataInput in) throws IOException {
>> match = in.readBoolean();
>> parents.readFields(in);
>> }
>>
>>
>>
>> And the error message is as follow:
>> Exception in thread "pool-6-thread-2" java.lang.RuntimeException:
>> java.lang.NullPointerException
>> at
>> org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:562)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.io.ArrayWritable.write(ArrayWritable.java:103)
>> at ProbMatch$ProbMatchVertex.writeState(ProbMatch.java:154)
>> at org.apache.hama.graph.Vertex.write(Vertex.java:311)
>> at
>> org.apache.hama.util.WritableUtils.unsafeSerialize(WritableUtils.java:55)
>> at
>> org.apache.hama.graph.MapVerticesInfo.put(MapVerticesInfo.java:64)
>> at
>> org.apache.hama.graph.GraphJobRunner.addVertex(GraphJobRunner.java:577)
>> at
>> org.apache.hama.graph.GraphJobRunner.access$300(GraphJobRunner.java:64)
>> at
>> org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>>
>>
>>
>> When I'm trying use the second method, I don't know how to read and write
>> an object array. I can use out.writeInt() method to write int. But when I
>> need to write an object. How can I do this? I'm so sorry that I'm not good
>> at Java. Here is the some code I'm trying to write.
>>
>>
>> static TextPair[] arr = new TextPair[1];
>>
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeBoolean(match);
>> out.writeInt(arr.length);
>> for(int i = 0; i < arr.length; i++) {
>> out.writeInt(arr[i].toString()); // Is this right?
>> }}
>>
>>
>> public void readState(DataInput in) throws IOException {
>> match = in.readBoolean();
>> int length = in.readInt();
>> for(int i=0;i<length;i++){
>> arr[i] = in.read // How can I read the textpair here?
>> }
>> }
>>
>>
>>
>> I'm very grateful to your help. Thanks again.
>> Best Regards, Ping Liu.
>>
>>
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Thu, Jan 28, 2016 11:51 AM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: RE: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Hi,
>>
>> You can use Hadoop built-in writable classes or own custom Writable.
>>
>> static ArrayWritable arr = new ArrayWritable(DoubleWritable.class);
>>
>> public void writeState(DataOutput out) throws IOException {
>> arr.write(out);
>> }
>>
>> Or,
>>
>> static int[] arr2 = new int[3];
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeInt(arr2.length);
>> for(int i = 0; i < arr2.length; i++) {
>> out.writeInt(arr2[i]);
>> }
>> }
>>
>> public void readState(DataInput in) throws IOException {
>> int size = in.readInt();
>> for(int i = 0; i < size; i++) {
>> arr2[i] = in.readInt();
>> }
>> }
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Thursday, January 28, 2016 11:16 AM
>> To: user
>> Subject: Re: RE: RE: RE: Do Hama support member member variable?
>>
>> Hi,
>> I still don't know how to use member variable. What's the input
>> parameter
>> DataOutput out in writeState() method? Could you please give me a example
>> to
>> use this method? Thanks a lot.
>> By the way, how can I deal with object member variable? For example,
>> the
>> member variable "private List parents" store the parents of the vertex. How
>> can I read and write this variable?
>> If you can take time to reply me, I will be very grateful to you.
>> Best Regard, Ping Liu.
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Wed, Jan 27, 2016 03:12 PM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Just FYI, Hama 0.7.1 release candidate is now available. Please feel free
>> to
>> use this, and it'd nice if you can let me know whether it works well with
>> you.
>>
>> Release tarball: http://people.apache.org/~edwardyoon/dist/0.7.1-RC1
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Wednesday, January 27, 2016 11:01 AM
>> To: user
>> Subject: Re: RE: RE: Do Hama support member member variable?
>>
>> I know. If I use the static variable like "private static boolean match", I
>> could get the right value of match too. Could I use static variable?
>> Thanks a lot.
>> Best regards. Ping Liu.
>>
>>
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Wed, Jan 27, 2016 09:39 AM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Basically Vertex object is writable, and we store the vertex objects in
>> serialized form. There are two purposes: 1) to reduce memory usage 2) to
>> write
>> on file system (checkpoint and recovery).
>>
>> So, you should use readState() and writeState() methods to save object
>> member
>> variables.
>>
>> Thanks!
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Wednesday, January 27, 2016 10:27 AM
>> To: user
>> Subject: Re: RE: Do Hama support member member variable?
>>
>> Hi,
>> Thank you very much. You helped me a lot. But I still don't know how to use
>> readState() and writeState() methods. What is the input parameter DataInput
>> in
>> and DataOutput out? And as far as I know, I can assign the value of member
>> variable directly in hama-0.6.3, such as match = true. Why I can't do this
>> in
>> Hama-0.7.0?
>> Waiting for your reply. Thanks.
>> Best Regards, Ping Liu.
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Wed, Jan 27, 2016 07:15 AM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: Do Hama support member member variable?
>>
>>
>>
>> Hi,
>>
>> You should use readState() and writeState() methods like below:
>>
>> public static class ProbMatchVertex extends Vertex<Text, NullWritable,
>> TriTextPair> {
>> private boolean match = false;
>>
>> public void readState(DataInput in) throws IOException {
>> match = in.readBoolean();
>> }
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeBoolean(match);
>> }
>>
>> ..
>>
>> }
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>> -----Original Message-----
>> From: 꼍행暾 [mailto:mailliuping@qq.com]
>> Sent: Tuesday, January 26, 2016 10:16 PM
>> To: user
>> Subject: Do Hama support member member variable?
>>
>> Hello,
>> I'm trying to run a graph job. But i have got some problems.
>> I want to use member variable in vertex class, the code is as follow.
>> I have changed the value of the member variable in one superstep. But when
>> I
>> use this member variable in next superstep, the value of this member
>> variable is still the defalult value. For example, "match" is the member
>> variable. I have changed the value of "match" to be ture in superstep 0,
>> but
>> when I print "match" in superstep 1, the result was "match: false".
>> Could anyone tell me why the value of member variable is changed?
>> Thanks very much.
>> Best wishes.
>>
>>
>> public static class ProbMatchVertex extends Vertex<Text,
>> NullWritable,
>> TriTextPair> {
>> private boolean match = false;
>>
>>
>> @Override
>> public void compute(Iterable<TriTextPair> messages) throws
>> IOException {
>> if (getSuperstepCount() == 0) {
>> match = true;
>> sendMessageToNeighbors(new
>> TriTextPair(getVertexID(),
>> getVertexLabel(), new
>> Text("")));
>> } else if(getSuperstepCount() == 1){
>> System.out.println("match:" + match);
>> parents = new ArrayList<TriTextPair>();
>> for(TriTextPair msg : messages){
>> parents.add(msg);
>> sendMessage(msg.getFirst(), new
>> TriTextPair(getVertexID(), getVertexLabel(), new Text("")));
>> }
>> }
>> }
>>
>>
>>
>>
>> Ping Liu
>
>
>
> --
> Best Regards, Edward J. Yoon
--
Best Regards, Edward J. Yoon
Re: RE: RE: RE: RE: RE: Do Hama support member member variable?
Posted by 步青云 <ma...@qq.com>.
Thank you. I'll use TRUNK version soon. And now I'm trying to read the source code of Hama. It's worth of learning. Reading the source code of hama could also help me to use hama better.
Best Regards, Ping Liu.
------------------ Original ------------------
From: "Edward J. Yoon";<ed...@samsung.com>;
Date: Wed, Feb 3, 2016 07:49 AM
To: "user"<us...@hama.apache.org>;
Subject: RE: RE: RE: RE: RE: Do Hama support member member variable?
Hi, 步?云
Please try to use TRUNK version. and you can refer the
https://github.com/apache/hama/blob/master/examples/src/test/java/org/apache/hama/examples/CustomVertexReadWriteStateTest.java
and
https://github.com/apache/hama/blob/master/ml/src/main/java/org/apache/hama/ml/kcore/KCoreVertex.java.
Both code contains custom read/writeState use case.
--
Best Regards, Edward J. Yoon
-----Original Message-----
From: Edward J. Yoon [mailto:edwardyoon@apache.org]
Sent: Thursday, January 28, 2016 8:33 PM
To: user@hama.apache.org
Subject: Re: RE: RE: RE: RE: Do Hama support member member variable?
I filed on https://issues.apache.org/jira/browse/HAMA-982
Once it's fixed, I'll let you know.
On Thu, Jan 28, 2016 at 8:11 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Wow, you find the bug. Thanks ;)
>
>> at
>> org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>
> When the framework assign vertices to the proper machine at initial
> phase, vertex objects are transferred in serialized form. At this step
> user defined code won't work correctly. I'll fix soon.
>
> Anyway, you should able to manage an array of TextPair objects like below:
>
> private TextPair[] test = new TextPair[1];
>
> public void readState(DataInput in) throws IOException {
> int size = in.readInt();
> test = new TextPair[size];
> for(int i = 0; i < size; i++) {
> test[i].readFields(in);
> }
> }
>
> public void writeState(DataOutput out) throws IOException {
> out.writeInt(test.length);
> for(int i = 0; i < test.length; i++) {
> test[i].write(out);
> }
> }
>
> Thanks.
>
>
> On Thu, Jan 28, 2016 at 6:45 PM, 步?云 <ma...@qq.com> wrote:
>> Thanks for your reply. You do help me a lot.
>> I have tried to use two methods. Some problem are still bothering me.
>> When I use the first method of using Hadoop built-in writable classes, I
>> get a NullPointerException meaning the parents is null， even though I have
>> initialize parents. The code is like this.
>> static ArrayWritable parents= new ArrayWritable(TextPair.class);
>>
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeBoolean(match);
>> parents.write(out);
>> }
>>
>>
>> public void readState(DataInput in) throws IOException {
>> match = in.readBoolean();
>> parents.readFields(in);
>> }
>>
>>
>>
>> And the error message is as follow:
>> Exception in thread "pool-6-thread-2" java.lang.RuntimeException:
>> java.lang.NullPointerException
>> at
>> org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:562)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.io.ArrayWritable.write(ArrayWritable.java:103)
>> at ProbMatch$ProbMatchVertex.writeState(ProbMatch.java:154)
>> at org.apache.hama.graph.Vertex.write(Vertex.java:311)
>> at
>> org.apache.hama.util.WritableUtils.unsafeSerialize(WritableUtils.java:55)
>> at
>> org.apache.hama.graph.MapVerticesInfo.put(MapVerticesInfo.java:64)
>> at
>> org.apache.hama.graph.GraphJobRunner.addVertex(GraphJobRunner.java:577)
>> at
>> org.apache.hama.graph.GraphJobRunner.access$300(GraphJobRunner.java:64)
>> at
>> org.apache.hama.graph.GraphJobRunner$Parser.run(GraphJobRunner.java:557)
>>
>>
>>
>> When I'm trying use the second method, I don't know how to read and write
>> an object array. I can use out.writeInt() method to write int. But when I
>> need to write an object. How can I do this? I'm so sorry that I'm not good
>> at Java. Here is the some code I'm trying to write.
>>
>>
>> static TextPair[] arr = new TextPair[1];
>>
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeBoolean(match);
>> out.writeInt(arr.length);
>> for(int i = 0; i < arr.length; i++) {
>> out.writeInt(arr[i].toString()); // Is this right?
>> }}
>>
>>
>> public void readState(DataInput in) throws IOException {
>> match = in.readBoolean();
>> int length = in.readInt();
>> for(int i=0;i<length;i++){
>> arr[i] = in.read // How can I read the textpair here?
>> }
>> }
>>
>>
>>
>> I'm very grateful to your help. Thanks again.
>> Best Regards, Ping Liu.
>>
>>
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Thu, Jan 28, 2016 11:51 AM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: RE: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Hi,
>>
>> You can use Hadoop built-in writable classes or own custom Writable.
>>
>> static ArrayWritable arr = new ArrayWritable(DoubleWritable.class);
>>
>> public void writeState(DataOutput out) throws IOException {
>> arr.write(out);
>> }
>>
>> Or,
>>
>> static int[] arr2 = new int[3];
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeInt(arr2.length);
>> for(int i = 0; i < arr2.length; i++) {
>> out.writeInt(arr2[i]);
>> }
>> }
>>
>> public void readState(DataInput in) throws IOException {
>> int size = in.readInt();
>> for(int i = 0; i < size; i++) {
>> arr2[i] = in.readInt();
>> }
>> }
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Thursday, January 28, 2016 11:16 AM
>> To: user
>> Subject: Re: RE: RE: RE: Do Hama support member member variable?
>>
>> Hi,
>> I still don't know how to use member variable. What's the input
>> parameter
>> DataOutput out in writeState() method? Could you please give me a example
>> to
>> use this method? Thanks a lot.
>> By the way, how can I deal with object member variable? For example,
>> the
>> member variable "private List parents" store the parents of the vertex. How
>> can I read and write this variable?
>> If you can take time to reply me, I will be very grateful to you.
>> Best Regard, Ping Liu.
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Wed, Jan 27, 2016 03:12 PM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Just FYI, Hama 0.7.1 release candidate is now available. Please feel free
>> to
>> use this, and it'd nice if you can let me know whether it works well with
>> you.
>>
>> Release tarball: http://people.apache.org/~edwardyoon/dist/0.7.1-RC1
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Wednesday, January 27, 2016 11:01 AM
>> To: user
>> Subject: Re: RE: RE: Do Hama support member member variable?
>>
>> I know. If I use the static variable like "private static boolean match", I
>> could get the right value of match too. Could I use static variable?
>> Thanks a lot.
>> Best regards. Ping Liu.
>>
>>
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Wed, Jan 27, 2016 09:39 AM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: RE: Do Hama support member member variable?
>>
>>
>>
>> Basically Vertex object is writable, and we store the vertex objects in
>> serialized form. There are two purposes: 1) to reduce memory usage 2) to
>> write
>> on file system (checkpoint and recovery).
>>
>> So, you should use readState() and writeState() methods to save object
>> member
>> variables.
>>
>> Thanks!
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>>
>> -----Original Message-----
>> From: 步?云 [mailto:mailliuping@qq.com]
>> Sent: Wednesday, January 27, 2016 10:27 AM
>> To: user
>> Subject: Re: RE: Do Hama support member member variable?
>>
>> Hi,
>> Thank you very much. You helped me a lot. But I still don't know how to use
>> readState() and writeState() methods. What is the input parameter DataInput
>> in
>> and DataOutput out? And as far as I know, I can assign the value of member
>> variable directly in hama-0.6.3, such as match = true. Why I can't do this
>> in
>> Hama-0.7.0?
>> Waiting for your reply. Thanks.
>> Best Regards, Ping Liu.
>>
>>
>> ------------------ Original ------------------
>> From: "Edward J. Yoon";<ed...@samsung.com>;
>> Date: Wed, Jan 27, 2016 07:15 AM
>> To: "user"<us...@hama.apache.org>;
>>
>> Subject: RE: Do Hama support member member variable?
>>
>>
>>
>> Hi,
>>
>> You should use readState() and writeState() methods like below:
>>
>> public static class ProbMatchVertex extends Vertex<Text, NullWritable,
>> TriTextPair> {
>> private boolean match = false;
>>
>> public void readState(DataInput in) throws IOException {
>> match = in.readBoolean();
>> }
>>
>> public void writeState(DataOutput out) throws IOException {
>> out.writeBoolean(match);
>> }
>>
>> ..
>>
>> }
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>> -----Original Message-----
>> From: 꼍행暾 [mailto:mailliuping@qq.com]
>> Sent: Tuesday, January 26, 2016 10:16 PM
>> To: user
>> Subject: Do Hama support member member variable?
>>
>> Hello,
>> I'm trying to run a graph job. But i have got some problems.
>> I want to use member variable in vertex class, the code is as follow.
>> I have changed the value of the member variable in one superstep. But when
>> I
>> use this member variable in next superstep, the value of this member
>> variable is still the defalult value. For example, "match" is the member
>> variable. I have changed the value of "match" to be ture in superstep 0,
>> but
>> when I print "match" in superstep 1, the result was "match: false".
>> Could anyone tell me why the value of member variable is changed?
>> Thanks very much.
>> Best wishes.
>>
>>
>> public static class ProbMatchVertex extends Vertex<Text,
>> NullWritable,
>> TriTextPair> {
>> private boolean match = false;
>>
>>
>> @Override
>> public void compute(Iterable<TriTextPair> messages) throws
>> IOException {
>> if (getSuperstepCount() == 0) {
>> match = true;
>> sendMessageToNeighbors(new
>> TriTextPair(getVertexID(),
>> getVertexLabel(), new
>> Text("")));
>> } else if(getSuperstepCount() == 1){
>> System.out.println("match:" + match);
>> parents = new ArrayList<TriTextPair>();
>> for(TriTextPair msg : messages){
>> parents.add(msg);
>> sendMessage(msg.getFirst(), new
>> TriTextPair(getVertexID(), getVertexLabel(), new Text("")));
>> }
>> }
>> }
>>
>>
>>
>>
>> Ping Liu
>
>
>
> --
> Best Regards, Edward J. Yoon
--
Best Regards, Edward J. Yoon