You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by maha <ma...@umail.ucsb.edu> on 2011/03/17 22:04:46 UTC
Writable Class with an Array
Hello,
I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.
I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :
public class twoIntegersArrayWritable implements Writable {
private int vectorSize;
private twoIntegers[] vector ;
/* CONSTRUCTORS */ ===================================================
twoIntegersArrayWritable () {}
twoIntegersArrayWritable (int size, twoIntegers[] vec) {
this.vectorSize = size;
this.vector = vec;
}
/* WRITE FUNCTION */ ===================================================
public void write(DataOutput out) {
out.writeInt(this.vectorSize);
for(int i=0;i<this.vectorSize;i++){
out.writeInt( vector[i]. getFirstIntger());
out.writeInt( vector[i]. getSecondIneger());
}
}
/* READ FUNCTION */ ===================================================
public void readFields(DataInput in) throws IOException {
this.vectorSize = in.readInt();
this.CreateNewVector (this.vectorSize);
for(int i=0;i< this.vectorSize;i++){
this.vector[i].setFirstInteger(in.readInt());
this.vector[i].setSecondInteger(in.readInt());
}
}
}
The errors I get:
java.lang.NullPointerException
at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)
Or it would work fine but array size is always 0. So it won't read the integer fields.
It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.
Maha
Re: Writable Class with an Array
Posted by David Rosenstrauch <da...@darose.net>.
I would try implementing this using an ArrayWritable, which contains an
array of IntWritables.
HTH,
DR
On 03/17/2011 05:04 PM, maha wrote:
> Hello,
>
> I'm stuck with this for two days now ...I found a previous post
> discussing this, but not with arrays.
>
> I know how to write Writable class with primitive type elements but
> this time I'm using an ARRAY of primitive type element, here it is in
> a simplified version for easy readability :
>
> public class twoIntegersArrayWritable implements Writable {
>
> private int vectorSize; private twoIntegers[] vector ;
>
> /* CONSTRUCTORS */
> ===================================================
>
> twoIntegersArrayWritable () {}
>
> twoIntegersArrayWritable (int size, twoIntegers[] vec) {
> this.vectorSize = size; this.vector = vec; }
>
> /* WRITE FUNCTION */
> ===================================================
>
> public void write(DataOutput out) {
>
> out.writeInt(this.vectorSize); for(int i=0;i<this.vectorSize;i++){
> out.writeInt( vector[i]. getFirstIntger()); out.writeInt( vector[i].
> getSecondIneger()); } }
>
> /* READ FUNCTION */
> ===================================================
>
> public void readFields(DataInput in) throws IOException {
> this.vectorSize = in.readInt(); this.CreateNewVector
> (this.vectorSize); for(int i=0;i< this.vectorSize;i++){
> this.vector[i].setFirstInteger(in.readInt());
> this.vector[i].setSecondInteger(in.readInt()); } } }
>
> The errors I get: java.lang.NullPointerException at
> twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
>
>
at
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
>
>
at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)
>
> Or it would work fine but array size is always 0. So it won't read
> the integer fields.
>
> It would be helpful if someone already has an example of using array
> in their Writable class. I appreciate any thought.
>
> Maha
>
>
Re: Writable Class with an Array
Posted by Matthew Foley <ma...@yahoo-inc.com>.
Maha and I went off-line and resolved this issue, which was just a Java programming error.
Cheers,
--Matt
On Mar 17, 2011, at 4:41 PM, maha wrote:
Thanks for your time Matt. Below are my clarifications ...
> There are three things missing from your simplified code needed to give a definitive answer, so I'll infer them as follows:
> - the specification of "twoIntegers"
> "twoIntegers" seems to be a tuple of ints with get/set accessors, so it is an object (array of primitives?) not a primitive, right?
public class twoIntegers () {
private int x;
private int y;
.....
}
so it's not an array. That's why I didn't use the ArrayWritable (although I tried but I didn't succeed).
> - in the exception "twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)", what line 46 refers to
> I'm betting it refers to the line "this.vector[i].setFirstInteger(in.readInt());"
Yes it is . After some investigation, it turns that 'in' is empty at this line, but it's because even though in the write function I have :
for(int i=0;i<this.vectorSize;i++)
out.writeInt( vector[i]. getFirstIntger());
out.writeInt( vector[i]. getSecondIneger());
but vectorSize is 0 ! so that's why nothing is written after vectorSize into DataOutputStream in the first place to be read later :( although the initialization of the class specifies clearly that the size is 10.
> - the implementation of CreateNewVector(size)
> It seems likely it just does a "new twoIntegers[size];"
Yes this is what I meant to do , because I wanted to make sure that my array is not of size 0, but it's still of size zero.
>
> If the above is correct, then the problem is that standard Java behavior is to fill the new array with nulls rather than with empty twoInteger objects. So in readFields(), when you enter the loop to read values for vector, the code "this.vector[0]" fetches a null object, and the continuation "this.vector[0].setFirstInteger()" causes a NullPointerException.
>
Do you mean I have to use Integer objects instead of primitive types ??
> You can fix this in CreateNewVector(), by explicitly allocating a new twoInteger object for each location in the "vector" array, or in the readFields() loop, whichever suits your program semantics better.
>
> --Matt
>
>
> On Mar 17, 2011, at 2:04 PM, maha wrote:
>
> Hello,
>
> I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.
>
> I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :
>
> public class twoIntegersArrayWritable implements Writable {
>
> private int vectorSize;
> private twoIntegers[] vector ;
>
> /* CONSTRUCTORS */ ===================================================
>
> twoIntegersArrayWritable () {}
>
> twoIntegersArrayWritable (int size, twoIntegers[] vec) {
> this.vectorSize = size;
> this.vector = vec;
> }
>
> /* WRITE FUNCTION */ ===================================================
>
> public void write(DataOutput out) {
>
> out.writeInt(this.vectorSize);
> for(int i=0;i<this.vectorSize;i++){
> out.writeInt( vector[i]. getFirstIntger());
> out.writeInt( vector[i]. getSecondIneger());
> }
> }
>
> /* READ FUNCTION */ ===================================================
>
> public void readFields(DataInput in) throws IOException {
>
> this.vectorSize = in.readInt();
> this.CreateNewVector (this.vectorSize);
> for(int i=0;i< this.vectorSize;i++){
> this.vector[i].setFirstInteger(in.readInt());
> this.vector[i].setSecondInteger(in.readInt());
> }
> }
> }
>
> The errors I get:
> java.lang.NullPointerException
> at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
> at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)
>
> Or it would work fine but array size is always 0. So it won't read the integer fields.
>
> It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.
>
> Maha
>
>
Re: Writable Class with an Array
Posted by maha <ma...@umail.ucsb.edu>.
Thanks for your time Matt. Below are my clarifications ...
> There are three things missing from your simplified code needed to give a definitive answer, so I'll infer them as follows:
> - the specification of "twoIntegers"
> "twoIntegers" seems to be a tuple of ints with get/set accessors, so it is an object (array of primitives?) not a primitive, right?
public class twoIntegers () {
private int x;
private int y;
.....
}
so it's not an array. That's why I didn't use the ArrayWritable (although I tried but I didn't succeed).
> - in the exception "twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)", what line 46 refers to
> I'm betting it refers to the line "this.vector[i].setFirstInteger(in.readInt());"
Yes it is . After some investigation, it turns that 'in' is empty at this line, but it's because even though in the write function I have :
for(int i=0;i<this.vectorSize;i++)
out.writeInt( vector[i]. getFirstIntger());
out.writeInt( vector[i]. getSecondIneger());
but vectorSize is 0 ! so that's why nothing is written after vectorSize into DataOutputStream in the first place to be read later :( although the initialization of the class specifies clearly that the size is 10.
> - the implementation of CreateNewVector(size)
> It seems likely it just does a "new twoIntegers[size];"
Yes this is what I meant to do , because I wanted to make sure that my array is not of size 0, but it's still of size zero.
>
> If the above is correct, then the problem is that standard Java behavior is to fill the new array with nulls rather than with empty twoInteger objects. So in readFields(), when you enter the loop to read values for vector, the code "this.vector[0]" fetches a null object, and the continuation "this.vector[0].setFirstInteger()" causes a NullPointerException.
>
Do you mean I have to use Integer objects instead of primitive types ??
> You can fix this in CreateNewVector(), by explicitly allocating a new twoInteger object for each location in the "vector" array, or in the readFields() loop, whichever suits your program semantics better.
>
> --Matt
>
>
> On Mar 17, 2011, at 2:04 PM, maha wrote:
>
> Hello,
>
> I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.
>
> I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :
>
> public class twoIntegersArrayWritable implements Writable {
>
> private int vectorSize;
> private twoIntegers[] vector ;
>
> /* CONSTRUCTORS */ ===================================================
>
> twoIntegersArrayWritable () {}
>
> twoIntegersArrayWritable (int size, twoIntegers[] vec) {
> this.vectorSize = size;
> this.vector = vec;
> }
>
> /* WRITE FUNCTION */ ===================================================
>
> public void write(DataOutput out) {
>
> out.writeInt(this.vectorSize);
> for(int i=0;i<this.vectorSize;i++){
> out.writeInt( vector[i]. getFirstIntger());
> out.writeInt( vector[i]. getSecondIneger());
> }
> }
>
> /* READ FUNCTION */ ===================================================
>
> public void readFields(DataInput in) throws IOException {
>
> this.vectorSize = in.readInt();
> this.CreateNewVector (this.vectorSize);
> for(int i=0;i< this.vectorSize;i++){
> this.vector[i].setFirstInteger(in.readInt());
> this.vector[i].setSecondInteger(in.readInt());
> }
> }
> }
>
> The errors I get:
> java.lang.NullPointerException
> at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
> at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)
>
> Or it would work fine but array size is always 0. So it won't read the integer fields.
>
> It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.
>
> Maha
>
>
Re: Writable Class with an Array
Posted by Matthew Foley <ma...@yahoo-inc.com>.
Hi Maha,
There are three things missing from your simplified code needed to give a definitive answer, so I'll infer them as follows:
- the specification of "twoIntegers"
"twoIntegers" seems to be a tuple of ints with get/set accessors, so it is an object (array of primitives?) not a primitive, right?
- in the exception "twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)", what line 46 refers to
I'm betting it refers to the line "this.vector[i].setFirstInteger(in.readInt());"
- the implementation of CreateNewVector(size)
It seems likely it just does a "new twoIntegers[size];"
If the above is correct, then the problem is that standard Java behavior is to fill the new array with nulls rather than with empty twoInteger objects. So in readFields(), when you enter the loop to read values for vector, the code "this.vector[0]" fetches a null object, and the continuation "this.vector[0].setFirstInteger()" causes a NullPointerException.
You can fix this in CreateNewVector(), by explicitly allocating a new twoInteger object for each location in the "vector" array, or in the readFields() loop, whichever suits your program semantics better.
--Matt
On Mar 17, 2011, at 2:04 PM, maha wrote:
Hello,
I'm stuck with this for two days now ...I found a previous post discussing this, but not with arrays.
I know how to write Writable class with primitive type elements but this time I'm using an ARRAY of primitive type element, here it is in a simplified version for easy readability :
public class twoIntegersArrayWritable implements Writable {
private int vectorSize;
private twoIntegers[] vector ;
/* CONSTRUCTORS */ ===================================================
twoIntegersArrayWritable () {}
twoIntegersArrayWritable (int size, twoIntegers[] vec) {
this.vectorSize = size;
this.vector = vec;
}
/* WRITE FUNCTION */ ===================================================
public void write(DataOutput out) {
out.writeInt(this.vectorSize);
for(int i=0;i<this.vectorSize;i++){
out.writeInt( vector[i]. getFirstIntger());
out.writeInt( vector[i]. getSecondIneger());
}
}
/* READ FUNCTION */ ===================================================
public void readFields(DataInput in) throws IOException {
this.vectorSize = in.readInt();
this.CreateNewVector (this.vectorSize);
for(int i=0;i< this.vectorSize;i++){
this.vector[i].setFirstInteger(in.readInt());
this.vector[i].setSecondInteger(in.readInt());
}
}
}
The errors I get:
java.lang.NullPointerException
at twoIntegersArrayWritable.readFields(twoIntegersArrayWritable.java:46)
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1751)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1879)
at SequenceFileReader_Integers.main(SequenceFileReader_Integers.java:49)
Or it would work fine but array size is always 0. So it won't read the integer fields.
It would be helpful if someone already has an example of using array in their Writable class. I appreciate any thought.
Maha