You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by unmesha sreeveni <un...@gmail.com> on 2013/10/31 12:03:07 UTC
Implementing a custom hadoop key and value - need Help
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
Again my output is empty. 3.Inorder to emit a key too as custom datatype
CompareTo is neccessary right .
4.so what should i include in that methods CompareTo,hashcode,equals and
what are these methods intended for.
Any Idea.Pls suggest a solution.
--
*Thanks & Regards*
*
*
Unmesha Sreeveni U.B*
*
*Junior Developer
*
*Amrita Center For Cyber Security
*
*
Amritapuri.
www.amrita.edu/cyber/
*
Re: Implementing a custom hadoop key and value - need Help
Posted by Steve Lewis <lo...@gmail.com>.
What is the purpose of using the matrix as a key - do you really expect
multiple values for the same key or not? Is there any reason to handle the
matrices in a specific order and if so what is the reason and what is the
order.
Unless you are dealing with thousands of millions of items with fairly
trivial operations I have had great success with serializing large objects
as strings and using Text - then deserializing them on the other end.
With VERY little cleverness most orderings can be converted to alphabetical
order of the test string.
It also means that all the code for testing ordering and
serialization/deserialization can be tested without using Hadoop. And you
can use one kind of mapper and reducer
On Mon, Nov 4, 2013 at 8:54 AM, unmesha sreeveni <un...@gmail.com>wrote:
> yes .By editing some of my code..... i am able to get my emitted matrix
> VALUE in reducer :) :) :) :).
> context.write(......, new MatrixWritable(Eval));
>
> But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
> CompareTo() method in my MatrixWritable class.
> can anyone suggest me.
>
>
> On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
>> beginners.
>>
>>
>> The second one is right :) .*But you wrote also: I need to emit a 2D
>> double array as key and value from mapper.*
>> *Means, you work with a k-v-pair *
>>
>> *KVP<Matrix,Matrix>*
>>
>> *There Matrix is 2-D matrix of double values.*
>>
>>
>> Yes i need to emit 2 matrices,1 key and the other is value.
>>
>> ie key -----> A*Atrans--------->after multiplication the result will be
>> a 2D array which is declared as double (matrix) lets say the result be
>> Matrix "*Ekey*"(double[][] Ekey)
>>
>> value ------> Atrans*D ---------> after multiplication the result will
>> be Matrix "*Eval*" (double[][] Eval).
>>
>> After tat i need to emit these matrix to reducer for further calculations.
>>
>> so in mapper
>> context.write(*Ekey*,*Eval*);
>>
>> reducer
>> i need to do further calculations with these Ekey nd Eval.
>>
>>
>> so i need to emit context.write(matrix,matrix).....for that i created
>> MatrixWritable class.
>>
>> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
>> 2.In reducer i gave iterable why becoz my key and value are matrices.That
>> y i gave them as iterable. IS nt that right?????
>> If wrong how to give the reducer signature.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(*Iterable<MatrixWritable>* key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> Usually a key is only one object, not an Iterable.
>>>
>>> To make things more clear:
>>>
>>> What is the exact k-v-pair you need in the Reducer?
>>>
>>> One matrix is the key, and a set of (two matrices together) are used as
>>> value in the Reducer? What I understood from your question is
>>> KVP<Matrix,Matrix[2]>
>>>
>>>
>>> *But you wrote also:* I need to emit a 2D double array as key and value
>>> from mapper.
>>> Means, you work with a k-v-pair
>>>
>>> KVP<Matrix,Matrix>
>>>
>>> There Matrix is 2-D matrix of double values.
>>>
>>> I suggest:
>>>
>>> 1.) Define the MR Data Flow.
>>> 2.) Build the custom types.
>>> 3.) Test the flow (no computation)
>>> 4.) Implement logic / computation
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>>
>>>> I tried with TwoDArrayWritable too.
>>>>
>>>> but i tried it by emitting only one value.
>>>>
>>>> row = E.length;
>>>> col = E[0].length;
>>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>>> // set values in myInnerArray
>>>> for (int k1 = 0; k1 < row; k1++) {
>>>> for(int j1=0;j1< col;j1++){
>>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>>
>>>> }
>>>> array.set(myInnerArray);
>>>> context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> this is also not working for me
>>>>
>>>> showing NullPointerException
>>>>
>>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>>> java.lang.Exception: java.lang.NullPointerException
>>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>>> Caused by: java.lang.NullPointerException
>>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>>> at edu.Mapper.map(Mapper.java:277)
>>>>
>>>>
>>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr Shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com> wrote:
>>>>>
>>>>>> @Amr shahin
>>>>>> this is my post from stackoverflow
>>>>>> but i am not getting any response.
>>>>>>
>>>>>>
>>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so how to emit these matrices from mapper.And the value
>>>>>> should be corresponding to the key itself.I think for that we need to use
>>>>>> WritableComparable.
>>>>>>
>>>>>>
>>>>>>
>>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>>
>>>>>> /**
>>>>>> * @param args
>>>>>> */
>>>>>> private double[][] value;
>>>>>>
>>>>>> public MatrixWritable() {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> set(new double[0][0]);
>>>>>> }
>>>>>>
>>>>>> public MatrixWritable(double[][] value) {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public void set(double[][] value) {
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public double[][] getValue() {
>>>>>> return value;
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void write(DataOutput out) throws IOException {
>>>>>> System.out.println("write");
>>>>>> int row=0;
>>>>>> int col=0;
>>>>>> for(int i=0; i<value.length;i++){
>>>>>> row = value.length;
>>>>>> for(int j=0; j<value[i].length; j++){
>>>>>> col = value[i].length;
>>>>>> }
>>>>>> }
>>>>>> out.writeInt(row);
>>>>>> out.writeInt(col);
>>>>>>
>>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>>
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>>
>>>>>> out.writeDouble(value[i][j]);
>>>>>> }
>>>>>> }
>>>>>> //priting array
>>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>>> }
>>>>>> System.out.println("");
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void readFields(DataInput in) throws IOException {
>>>>>> int row = in.readInt();
>>>>>> int col = in.readInt();
>>>>>>
>>>>>> double[][] value = new double[row][col];
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>> value[i][j] = in.readDouble();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public int hashCode() {
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public boolean equals(Object o) {
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> @Override
>>>>>> public int compareTo(MatrixWritable o) {
>>>>>> // TODO Auto-generated method stub
>>>>>> return 0;
>>>>>> }
>>>>>> @Override
>>>>>> public String toString() {
>>>>>>
>>>>>> return Arrays.toString(value);
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> I wrote matrix write,readfields and toString.
>>>>>>
>>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>>
>>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>>
>>>>>> public class MyReducer extends
>>>>>>
>>>>>>
>>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>
>>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>>> Iterable<MatrixWritable> values, Context context){
>>>>>> for(MatrixWritable c : values){
>>>>>>
>>>>>> System.out.println("print value "+c.toString());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>>> i also needed to print my data asmatrix so i tried
>>>>>>
>>>>>> public String toString() {
>>>>>>
>>>>>> String separator = ", ";
>>>>>> StringBuffer result = new StringBuffer();
>>>>>>
>>>>>> // iterate over the first dimension
>>>>>> for (int i = 0; i < value.length; i++) {
>>>>>> // iterate over the second dimension
>>>>>> for(int j = 0; j < value[i].length; j++){
>>>>>> result.append(value[i][j]);
>>>>>> System.out.print(value[i][j]);
>>>>>> result.append(separator);
>>>>>> }
>>>>>> // remove the last separator
>>>>>> result.setLength(result.length() - separator.length());
>>>>>> // add a line break.
>>>>>> result.append("\n");
>>>>>> }
>>>>>>
>>>>>>
>>>>>> return result.toString();
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Can you share the code?
>>>>>>>
>>>>>>> sent from mobile
>>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>>> Amr Shahin , i refered "
>>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Check this out:
>>>>>>>>>
>>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>>> It shows how to create a customer writable. If you have "hadoop
>>>>>>>>> the
>>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>>> data
>>>>>>>>> types.
>>>>>>>>>
>>>>>>>>> Happy Halloween
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>>
>>>>>>>>> > this is my post from stackoverflow
>>>>>>>>> > but i am not getting any response.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>>> mapper.There are
>>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>>> custom
>>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>>> the matrix
>>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>>> the value
>>>>>>>>> > of
>>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be
>>>>>>>>> a matrix.so
>>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>>> > WritableComparable.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > public class MatrixWritable implements
>>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>>> >
>>>>>>>>> > /**
>>>>>>>>> > * @param args
>>>>>>>>> > */
>>>>>>>>> > private double[][] value;
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable() {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > set(new double[0][0]);
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public void set(double[][] value) {
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public double[][] getValue() {
>>>>>>>>> > return value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>>> > System.out.println("write");
>>>>>>>>> > int row=0;
>>>>>>>>> > int col=0;
>>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>>> > row = value.length;
>>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>>> > col = value[i].length;
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > out.writeInt(row);
>>>>>>>>> > out.writeInt(col);
>>>>>>>>> >
>>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>>> "+row+":"+col);
>>>>>>>>> >
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> >
>>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > //priting array
>>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>>> > }
>>>>>>>>> > System.out.println("");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>>> > int row = in.readInt();
>>>>>>>>> > int col = in.readInt();
>>>>>>>>> >
>>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int hashCode() {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public boolean equals(Object o) {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>>> > return 0;
>>>>>>>>> > }
>>>>>>>>> > @Override
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > return Arrays.toString(value);
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>>> >
>>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>>> >
>>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>>> with
>>>>>>>>> > emiting only custom value- for checking)
>>>>>>>>> >
>>>>>>>>> > public class MyReducer extends
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>>> >
>>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>>> >
>>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>>> toString()
>>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>>> wrong.and i
>>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>>> >
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > String separator = ", ";
>>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>>> >
>>>>>>>>> > // iterate over the first dimension
>>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>>> > // iterate over the second dimension
>>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>>> > result.append(value[i][j]);
>>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>>> > result.append(separator);
>>>>>>>>> > }
>>>>>>>>> > // remove the last separator
>>>>>>>>> > result.setLength(result.length() -
>>>>>>>>> separator.length());
>>>>>>>>> > // add a line break.
>>>>>>>>> > result.append("\n");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > return result.toString();
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>>> datatype
>>>>>>>>> > CompareTo is neccessary right .
>>>>>>>>> >
>>>>>>>>> > 4.so what should i include in that methods
>>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>>> > what are these methods intended for.
>>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > *Thanks & Regards*
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>>> > *
>>>>>>>>> > *Junior Developer
>>>>>>>>> > *
>>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Amritapuri.
>>>>>>>>> >
>>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>>> > *
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Thanks & Regards*
>>>>>>>>
>>>>>>>> Unmesha Sreeveni U.B
>>>>>>>>
>>>>>>>> *Junior Developer*
>>>>>>>>
>>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>>
>>>>>>>>
>>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Implementing a custom hadoop key and value - need Help
Posted by Steve Lewis <lo...@gmail.com>.
What is the purpose of using the matrix as a key - do you really expect
multiple values for the same key or not? Is there any reason to handle the
matrices in a specific order and if so what is the reason and what is the
order.
Unless you are dealing with thousands of millions of items with fairly
trivial operations I have had great success with serializing large objects
as strings and using Text - then deserializing them on the other end.
With VERY little cleverness most orderings can be converted to alphabetical
order of the test string.
It also means that all the code for testing ordering and
serialization/deserialization can be tested without using Hadoop. And you
can use one kind of mapper and reducer
On Mon, Nov 4, 2013 at 8:54 AM, unmesha sreeveni <un...@gmail.com>wrote:
> yes .By editing some of my code..... i am able to get my emitted matrix
> VALUE in reducer :) :) :) :).
> context.write(......, new MatrixWritable(Eval));
>
> But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
> CompareTo() method in my MatrixWritable class.
> can anyone suggest me.
>
>
> On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
>> beginners.
>>
>>
>> The second one is right :) .*But you wrote also: I need to emit a 2D
>> double array as key and value from mapper.*
>> *Means, you work with a k-v-pair *
>>
>> *KVP<Matrix,Matrix>*
>>
>> *There Matrix is 2-D matrix of double values.*
>>
>>
>> Yes i need to emit 2 matrices,1 key and the other is value.
>>
>> ie key -----> A*Atrans--------->after multiplication the result will be
>> a 2D array which is declared as double (matrix) lets say the result be
>> Matrix "*Ekey*"(double[][] Ekey)
>>
>> value ------> Atrans*D ---------> after multiplication the result will
>> be Matrix "*Eval*" (double[][] Eval).
>>
>> After tat i need to emit these matrix to reducer for further calculations.
>>
>> so in mapper
>> context.write(*Ekey*,*Eval*);
>>
>> reducer
>> i need to do further calculations with these Ekey nd Eval.
>>
>>
>> so i need to emit context.write(matrix,matrix).....for that i created
>> MatrixWritable class.
>>
>> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
>> 2.In reducer i gave iterable why becoz my key and value are matrices.That
>> y i gave them as iterable. IS nt that right?????
>> If wrong how to give the reducer signature.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(*Iterable<MatrixWritable>* key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> Usually a key is only one object, not an Iterable.
>>>
>>> To make things more clear:
>>>
>>> What is the exact k-v-pair you need in the Reducer?
>>>
>>> One matrix is the key, and a set of (two matrices together) are used as
>>> value in the Reducer? What I understood from your question is
>>> KVP<Matrix,Matrix[2]>
>>>
>>>
>>> *But you wrote also:* I need to emit a 2D double array as key and value
>>> from mapper.
>>> Means, you work with a k-v-pair
>>>
>>> KVP<Matrix,Matrix>
>>>
>>> There Matrix is 2-D matrix of double values.
>>>
>>> I suggest:
>>>
>>> 1.) Define the MR Data Flow.
>>> 2.) Build the custom types.
>>> 3.) Test the flow (no computation)
>>> 4.) Implement logic / computation
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>>
>>>> I tried with TwoDArrayWritable too.
>>>>
>>>> but i tried it by emitting only one value.
>>>>
>>>> row = E.length;
>>>> col = E[0].length;
>>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>>> // set values in myInnerArray
>>>> for (int k1 = 0; k1 < row; k1++) {
>>>> for(int j1=0;j1< col;j1++){
>>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>>
>>>> }
>>>> array.set(myInnerArray);
>>>> context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> this is also not working for me
>>>>
>>>> showing NullPointerException
>>>>
>>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>>> java.lang.Exception: java.lang.NullPointerException
>>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>>> Caused by: java.lang.NullPointerException
>>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>>> at edu.Mapper.map(Mapper.java:277)
>>>>
>>>>
>>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr Shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com> wrote:
>>>>>
>>>>>> @Amr shahin
>>>>>> this is my post from stackoverflow
>>>>>> but i am not getting any response.
>>>>>>
>>>>>>
>>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so how to emit these matrices from mapper.And the value
>>>>>> should be corresponding to the key itself.I think for that we need to use
>>>>>> WritableComparable.
>>>>>>
>>>>>>
>>>>>>
>>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>>
>>>>>> /**
>>>>>> * @param args
>>>>>> */
>>>>>> private double[][] value;
>>>>>>
>>>>>> public MatrixWritable() {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> set(new double[0][0]);
>>>>>> }
>>>>>>
>>>>>> public MatrixWritable(double[][] value) {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public void set(double[][] value) {
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public double[][] getValue() {
>>>>>> return value;
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void write(DataOutput out) throws IOException {
>>>>>> System.out.println("write");
>>>>>> int row=0;
>>>>>> int col=0;
>>>>>> for(int i=0; i<value.length;i++){
>>>>>> row = value.length;
>>>>>> for(int j=0; j<value[i].length; j++){
>>>>>> col = value[i].length;
>>>>>> }
>>>>>> }
>>>>>> out.writeInt(row);
>>>>>> out.writeInt(col);
>>>>>>
>>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>>
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>>
>>>>>> out.writeDouble(value[i][j]);
>>>>>> }
>>>>>> }
>>>>>> //priting array
>>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>>> }
>>>>>> System.out.println("");
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void readFields(DataInput in) throws IOException {
>>>>>> int row = in.readInt();
>>>>>> int col = in.readInt();
>>>>>>
>>>>>> double[][] value = new double[row][col];
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>> value[i][j] = in.readDouble();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public int hashCode() {
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public boolean equals(Object o) {
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> @Override
>>>>>> public int compareTo(MatrixWritable o) {
>>>>>> // TODO Auto-generated method stub
>>>>>> return 0;
>>>>>> }
>>>>>> @Override
>>>>>> public String toString() {
>>>>>>
>>>>>> return Arrays.toString(value);
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> I wrote matrix write,readfields and toString.
>>>>>>
>>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>>
>>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>>
>>>>>> public class MyReducer extends
>>>>>>
>>>>>>
>>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>
>>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>>> Iterable<MatrixWritable> values, Context context){
>>>>>> for(MatrixWritable c : values){
>>>>>>
>>>>>> System.out.println("print value "+c.toString());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>>> i also needed to print my data asmatrix so i tried
>>>>>>
>>>>>> public String toString() {
>>>>>>
>>>>>> String separator = ", ";
>>>>>> StringBuffer result = new StringBuffer();
>>>>>>
>>>>>> // iterate over the first dimension
>>>>>> for (int i = 0; i < value.length; i++) {
>>>>>> // iterate over the second dimension
>>>>>> for(int j = 0; j < value[i].length; j++){
>>>>>> result.append(value[i][j]);
>>>>>> System.out.print(value[i][j]);
>>>>>> result.append(separator);
>>>>>> }
>>>>>> // remove the last separator
>>>>>> result.setLength(result.length() - separator.length());
>>>>>> // add a line break.
>>>>>> result.append("\n");
>>>>>> }
>>>>>>
>>>>>>
>>>>>> return result.toString();
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Can you share the code?
>>>>>>>
>>>>>>> sent from mobile
>>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>>> Amr Shahin , i refered "
>>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Check this out:
>>>>>>>>>
>>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>>> It shows how to create a customer writable. If you have "hadoop
>>>>>>>>> the
>>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>>> data
>>>>>>>>> types.
>>>>>>>>>
>>>>>>>>> Happy Halloween
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>>
>>>>>>>>> > this is my post from stackoverflow
>>>>>>>>> > but i am not getting any response.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>>> mapper.There are
>>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>>> custom
>>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>>> the matrix
>>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>>> the value
>>>>>>>>> > of
>>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be
>>>>>>>>> a matrix.so
>>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>>> > WritableComparable.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > public class MatrixWritable implements
>>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>>> >
>>>>>>>>> > /**
>>>>>>>>> > * @param args
>>>>>>>>> > */
>>>>>>>>> > private double[][] value;
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable() {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > set(new double[0][0]);
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public void set(double[][] value) {
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public double[][] getValue() {
>>>>>>>>> > return value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>>> > System.out.println("write");
>>>>>>>>> > int row=0;
>>>>>>>>> > int col=0;
>>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>>> > row = value.length;
>>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>>> > col = value[i].length;
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > out.writeInt(row);
>>>>>>>>> > out.writeInt(col);
>>>>>>>>> >
>>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>>> "+row+":"+col);
>>>>>>>>> >
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> >
>>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > //priting array
>>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>>> > }
>>>>>>>>> > System.out.println("");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>>> > int row = in.readInt();
>>>>>>>>> > int col = in.readInt();
>>>>>>>>> >
>>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int hashCode() {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public boolean equals(Object o) {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>>> > return 0;
>>>>>>>>> > }
>>>>>>>>> > @Override
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > return Arrays.toString(value);
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>>> >
>>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>>> >
>>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>>> with
>>>>>>>>> > emiting only custom value- for checking)
>>>>>>>>> >
>>>>>>>>> > public class MyReducer extends
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>>> >
>>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>>> >
>>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>>> toString()
>>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>>> wrong.and i
>>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>>> >
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > String separator = ", ";
>>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>>> >
>>>>>>>>> > // iterate over the first dimension
>>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>>> > // iterate over the second dimension
>>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>>> > result.append(value[i][j]);
>>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>>> > result.append(separator);
>>>>>>>>> > }
>>>>>>>>> > // remove the last separator
>>>>>>>>> > result.setLength(result.length() -
>>>>>>>>> separator.length());
>>>>>>>>> > // add a line break.
>>>>>>>>> > result.append("\n");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > return result.toString();
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>>> datatype
>>>>>>>>> > CompareTo is neccessary right .
>>>>>>>>> >
>>>>>>>>> > 4.so what should i include in that methods
>>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>>> > what are these methods intended for.
>>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > *Thanks & Regards*
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>>> > *
>>>>>>>>> > *Junior Developer
>>>>>>>>> > *
>>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Amritapuri.
>>>>>>>>> >
>>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>>> > *
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Thanks & Regards*
>>>>>>>>
>>>>>>>> Unmesha Sreeveni U.B
>>>>>>>>
>>>>>>>> *Junior Developer*
>>>>>>>>
>>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>>
>>>>>>>>
>>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Implementing a custom hadoop key and value - need Help
Posted by Steve Lewis <lo...@gmail.com>.
What is the purpose of using the matrix as a key - do you really expect
multiple values for the same key or not? Is there any reason to handle the
matrices in a specific order and if so what is the reason and what is the
order.
Unless you are dealing with thousands of millions of items with fairly
trivial operations I have had great success with serializing large objects
as strings and using Text - then deserializing them on the other end.
With VERY little cleverness most orderings can be converted to alphabetical
order of the test string.
It also means that all the code for testing ordering and
serialization/deserialization can be tested without using Hadoop. And you
can use one kind of mapper and reducer
On Mon, Nov 4, 2013 at 8:54 AM, unmesha sreeveni <un...@gmail.com>wrote:
> yes .By editing some of my code..... i am able to get my emitted matrix
> VALUE in reducer :) :) :) :).
> context.write(......, new MatrixWritable(Eval));
>
> But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
> CompareTo() method in my MatrixWritable class.
> can anyone suggest me.
>
>
> On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
>> beginners.
>>
>>
>> The second one is right :) .*But you wrote also: I need to emit a 2D
>> double array as key and value from mapper.*
>> *Means, you work with a k-v-pair *
>>
>> *KVP<Matrix,Matrix>*
>>
>> *There Matrix is 2-D matrix of double values.*
>>
>>
>> Yes i need to emit 2 matrices,1 key and the other is value.
>>
>> ie key -----> A*Atrans--------->after multiplication the result will be
>> a 2D array which is declared as double (matrix) lets say the result be
>> Matrix "*Ekey*"(double[][] Ekey)
>>
>> value ------> Atrans*D ---------> after multiplication the result will
>> be Matrix "*Eval*" (double[][] Eval).
>>
>> After tat i need to emit these matrix to reducer for further calculations.
>>
>> so in mapper
>> context.write(*Ekey*,*Eval*);
>>
>> reducer
>> i need to do further calculations with these Ekey nd Eval.
>>
>>
>> so i need to emit context.write(matrix,matrix).....for that i created
>> MatrixWritable class.
>>
>> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
>> 2.In reducer i gave iterable why becoz my key and value are matrices.That
>> y i gave them as iterable. IS nt that right?????
>> If wrong how to give the reducer signature.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(*Iterable<MatrixWritable>* key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> Usually a key is only one object, not an Iterable.
>>>
>>> To make things more clear:
>>>
>>> What is the exact k-v-pair you need in the Reducer?
>>>
>>> One matrix is the key, and a set of (two matrices together) are used as
>>> value in the Reducer? What I understood from your question is
>>> KVP<Matrix,Matrix[2]>
>>>
>>>
>>> *But you wrote also:* I need to emit a 2D double array as key and value
>>> from mapper.
>>> Means, you work with a k-v-pair
>>>
>>> KVP<Matrix,Matrix>
>>>
>>> There Matrix is 2-D matrix of double values.
>>>
>>> I suggest:
>>>
>>> 1.) Define the MR Data Flow.
>>> 2.) Build the custom types.
>>> 3.) Test the flow (no computation)
>>> 4.) Implement logic / computation
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>>
>>>> I tried with TwoDArrayWritable too.
>>>>
>>>> but i tried it by emitting only one value.
>>>>
>>>> row = E.length;
>>>> col = E[0].length;
>>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>>> // set values in myInnerArray
>>>> for (int k1 = 0; k1 < row; k1++) {
>>>> for(int j1=0;j1< col;j1++){
>>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>>
>>>> }
>>>> array.set(myInnerArray);
>>>> context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> this is also not working for me
>>>>
>>>> showing NullPointerException
>>>>
>>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>>> java.lang.Exception: java.lang.NullPointerException
>>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>>> Caused by: java.lang.NullPointerException
>>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>>> at edu.Mapper.map(Mapper.java:277)
>>>>
>>>>
>>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr Shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com> wrote:
>>>>>
>>>>>> @Amr shahin
>>>>>> this is my post from stackoverflow
>>>>>> but i am not getting any response.
>>>>>>
>>>>>>
>>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so how to emit these matrices from mapper.And the value
>>>>>> should be corresponding to the key itself.I think for that we need to use
>>>>>> WritableComparable.
>>>>>>
>>>>>>
>>>>>>
>>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>>
>>>>>> /**
>>>>>> * @param args
>>>>>> */
>>>>>> private double[][] value;
>>>>>>
>>>>>> public MatrixWritable() {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> set(new double[0][0]);
>>>>>> }
>>>>>>
>>>>>> public MatrixWritable(double[][] value) {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public void set(double[][] value) {
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public double[][] getValue() {
>>>>>> return value;
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void write(DataOutput out) throws IOException {
>>>>>> System.out.println("write");
>>>>>> int row=0;
>>>>>> int col=0;
>>>>>> for(int i=0; i<value.length;i++){
>>>>>> row = value.length;
>>>>>> for(int j=0; j<value[i].length; j++){
>>>>>> col = value[i].length;
>>>>>> }
>>>>>> }
>>>>>> out.writeInt(row);
>>>>>> out.writeInt(col);
>>>>>>
>>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>>
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>>
>>>>>> out.writeDouble(value[i][j]);
>>>>>> }
>>>>>> }
>>>>>> //priting array
>>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>>> }
>>>>>> System.out.println("");
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void readFields(DataInput in) throws IOException {
>>>>>> int row = in.readInt();
>>>>>> int col = in.readInt();
>>>>>>
>>>>>> double[][] value = new double[row][col];
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>> value[i][j] = in.readDouble();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public int hashCode() {
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public boolean equals(Object o) {
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> @Override
>>>>>> public int compareTo(MatrixWritable o) {
>>>>>> // TODO Auto-generated method stub
>>>>>> return 0;
>>>>>> }
>>>>>> @Override
>>>>>> public String toString() {
>>>>>>
>>>>>> return Arrays.toString(value);
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> I wrote matrix write,readfields and toString.
>>>>>>
>>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>>
>>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>>
>>>>>> public class MyReducer extends
>>>>>>
>>>>>>
>>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>
>>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>>> Iterable<MatrixWritable> values, Context context){
>>>>>> for(MatrixWritable c : values){
>>>>>>
>>>>>> System.out.println("print value "+c.toString());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>>> i also needed to print my data asmatrix so i tried
>>>>>>
>>>>>> public String toString() {
>>>>>>
>>>>>> String separator = ", ";
>>>>>> StringBuffer result = new StringBuffer();
>>>>>>
>>>>>> // iterate over the first dimension
>>>>>> for (int i = 0; i < value.length; i++) {
>>>>>> // iterate over the second dimension
>>>>>> for(int j = 0; j < value[i].length; j++){
>>>>>> result.append(value[i][j]);
>>>>>> System.out.print(value[i][j]);
>>>>>> result.append(separator);
>>>>>> }
>>>>>> // remove the last separator
>>>>>> result.setLength(result.length() - separator.length());
>>>>>> // add a line break.
>>>>>> result.append("\n");
>>>>>> }
>>>>>>
>>>>>>
>>>>>> return result.toString();
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Can you share the code?
>>>>>>>
>>>>>>> sent from mobile
>>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>>> Amr Shahin , i refered "
>>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Check this out:
>>>>>>>>>
>>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>>> It shows how to create a customer writable. If you have "hadoop
>>>>>>>>> the
>>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>>> data
>>>>>>>>> types.
>>>>>>>>>
>>>>>>>>> Happy Halloween
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>>
>>>>>>>>> > this is my post from stackoverflow
>>>>>>>>> > but i am not getting any response.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>>> mapper.There are
>>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>>> custom
>>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>>> the matrix
>>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>>> the value
>>>>>>>>> > of
>>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be
>>>>>>>>> a matrix.so
>>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>>> > WritableComparable.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > public class MatrixWritable implements
>>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>>> >
>>>>>>>>> > /**
>>>>>>>>> > * @param args
>>>>>>>>> > */
>>>>>>>>> > private double[][] value;
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable() {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > set(new double[0][0]);
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public void set(double[][] value) {
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public double[][] getValue() {
>>>>>>>>> > return value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>>> > System.out.println("write");
>>>>>>>>> > int row=0;
>>>>>>>>> > int col=0;
>>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>>> > row = value.length;
>>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>>> > col = value[i].length;
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > out.writeInt(row);
>>>>>>>>> > out.writeInt(col);
>>>>>>>>> >
>>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>>> "+row+":"+col);
>>>>>>>>> >
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> >
>>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > //priting array
>>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>>> > }
>>>>>>>>> > System.out.println("");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>>> > int row = in.readInt();
>>>>>>>>> > int col = in.readInt();
>>>>>>>>> >
>>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int hashCode() {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public boolean equals(Object o) {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>>> > return 0;
>>>>>>>>> > }
>>>>>>>>> > @Override
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > return Arrays.toString(value);
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>>> >
>>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>>> >
>>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>>> with
>>>>>>>>> > emiting only custom value- for checking)
>>>>>>>>> >
>>>>>>>>> > public class MyReducer extends
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>>> >
>>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>>> >
>>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>>> toString()
>>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>>> wrong.and i
>>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>>> >
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > String separator = ", ";
>>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>>> >
>>>>>>>>> > // iterate over the first dimension
>>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>>> > // iterate over the second dimension
>>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>>> > result.append(value[i][j]);
>>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>>> > result.append(separator);
>>>>>>>>> > }
>>>>>>>>> > // remove the last separator
>>>>>>>>> > result.setLength(result.length() -
>>>>>>>>> separator.length());
>>>>>>>>> > // add a line break.
>>>>>>>>> > result.append("\n");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > return result.toString();
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>>> datatype
>>>>>>>>> > CompareTo is neccessary right .
>>>>>>>>> >
>>>>>>>>> > 4.so what should i include in that methods
>>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>>> > what are these methods intended for.
>>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > *Thanks & Regards*
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>>> > *
>>>>>>>>> > *Junior Developer
>>>>>>>>> > *
>>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Amritapuri.
>>>>>>>>> >
>>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>>> > *
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Thanks & Regards*
>>>>>>>>
>>>>>>>> Unmesha Sreeveni U.B
>>>>>>>>
>>>>>>>> *Junior Developer*
>>>>>>>>
>>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>>
>>>>>>>>
>>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Implementing a custom hadoop key and value - need Help
Posted by Steve Lewis <lo...@gmail.com>.
What is the purpose of using the matrix as a key - do you really expect
multiple values for the same key or not? Is there any reason to handle the
matrices in a specific order and if so what is the reason and what is the
order.
Unless you are dealing with thousands of millions of items with fairly
trivial operations I have had great success with serializing large objects
as strings and using Text - then deserializing them on the other end.
With VERY little cleverness most orderings can be converted to alphabetical
order of the test string.
It also means that all the code for testing ordering and
serialization/deserialization can be tested without using Hadoop. And you
can use one kind of mapper and reducer
On Mon, Nov 4, 2013 at 8:54 AM, unmesha sreeveni <un...@gmail.com>wrote:
> yes .By editing some of my code..... i am able to get my emitted matrix
> VALUE in reducer :) :) :) :).
> context.write(......, new MatrixWritable(Eval));
>
> But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
> CompareTo() method in my MatrixWritable class.
> can anyone suggest me.
>
>
> On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
>> beginners.
>>
>>
>> The second one is right :) .*But you wrote also: I need to emit a 2D
>> double array as key and value from mapper.*
>> *Means, you work with a k-v-pair *
>>
>> *KVP<Matrix,Matrix>*
>>
>> *There Matrix is 2-D matrix of double values.*
>>
>>
>> Yes i need to emit 2 matrices,1 key and the other is value.
>>
>> ie key -----> A*Atrans--------->after multiplication the result will be
>> a 2D array which is declared as double (matrix) lets say the result be
>> Matrix "*Ekey*"(double[][] Ekey)
>>
>> value ------> Atrans*D ---------> after multiplication the result will
>> be Matrix "*Eval*" (double[][] Eval).
>>
>> After tat i need to emit these matrix to reducer for further calculations.
>>
>> so in mapper
>> context.write(*Ekey*,*Eval*);
>>
>> reducer
>> i need to do further calculations with these Ekey nd Eval.
>>
>>
>> so i need to emit context.write(matrix,matrix).....for that i created
>> MatrixWritable class.
>>
>> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
>> 2.In reducer i gave iterable why becoz my key and value are matrices.That
>> y i gave them as iterable. IS nt that right?????
>> If wrong how to give the reducer signature.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(*Iterable<MatrixWritable>* key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> Usually a key is only one object, not an Iterable.
>>>
>>> To make things more clear:
>>>
>>> What is the exact k-v-pair you need in the Reducer?
>>>
>>> One matrix is the key, and a set of (two matrices together) are used as
>>> value in the Reducer? What I understood from your question is
>>> KVP<Matrix,Matrix[2]>
>>>
>>>
>>> *But you wrote also:* I need to emit a 2D double array as key and value
>>> from mapper.
>>> Means, you work with a k-v-pair
>>>
>>> KVP<Matrix,Matrix>
>>>
>>> There Matrix is 2-D matrix of double values.
>>>
>>> I suggest:
>>>
>>> 1.) Define the MR Data Flow.
>>> 2.) Build the custom types.
>>> 3.) Test the flow (no computation)
>>> 4.) Implement logic / computation
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>>
>>>> I tried with TwoDArrayWritable too.
>>>>
>>>> but i tried it by emitting only one value.
>>>>
>>>> row = E.length;
>>>> col = E[0].length;
>>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>>> // set values in myInnerArray
>>>> for (int k1 = 0; k1 < row; k1++) {
>>>> for(int j1=0;j1< col;j1++){
>>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>>
>>>> }
>>>> array.set(myInnerArray);
>>>> context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> this is also not working for me
>>>>
>>>> showing NullPointerException
>>>>
>>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>>> java.lang.Exception: java.lang.NullPointerException
>>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>>> Caused by: java.lang.NullPointerException
>>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>>> at edu.Mapper.map(Mapper.java:277)
>>>>
>>>>
>>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr Shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com> wrote:
>>>>>
>>>>>> @Amr shahin
>>>>>> this is my post from stackoverflow
>>>>>> but i am not getting any response.
>>>>>>
>>>>>>
>>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so how to emit these matrices from mapper.And the value
>>>>>> should be corresponding to the key itself.I think for that we need to use
>>>>>> WritableComparable.
>>>>>>
>>>>>>
>>>>>>
>>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>>
>>>>>> /**
>>>>>> * @param args
>>>>>> */
>>>>>> private double[][] value;
>>>>>>
>>>>>> public MatrixWritable() {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> set(new double[0][0]);
>>>>>> }
>>>>>>
>>>>>> public MatrixWritable(double[][] value) {
>>>>>> // TODO Auto-generated constructor stub
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public void set(double[][] value) {
>>>>>> this.value = value;
>>>>>> }
>>>>>>
>>>>>> public double[][] getValue() {
>>>>>> return value;
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void write(DataOutput out) throws IOException {
>>>>>> System.out.println("write");
>>>>>> int row=0;
>>>>>> int col=0;
>>>>>> for(int i=0; i<value.length;i++){
>>>>>> row = value.length;
>>>>>> for(int j=0; j<value[i].length; j++){
>>>>>> col = value[i].length;
>>>>>> }
>>>>>> }
>>>>>> out.writeInt(row);
>>>>>> out.writeInt(col);
>>>>>>
>>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>>
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>>
>>>>>> out.writeDouble(value[i][j]);
>>>>>> }
>>>>>> }
>>>>>> //priting array
>>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>>> }
>>>>>> System.out.println("");
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public void readFields(DataInput in) throws IOException {
>>>>>> int row = in.readInt();
>>>>>> int col = in.readInt();
>>>>>>
>>>>>> double[][] value = new double[row][col];
>>>>>> for(int i=0;i<row ; i++){
>>>>>> for(int j= 0 ; j< col;j++){
>>>>>> value[i][j] = in.readDouble();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public int hashCode() {
>>>>>>
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public boolean equals(Object o) {
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> @Override
>>>>>> public int compareTo(MatrixWritable o) {
>>>>>> // TODO Auto-generated method stub
>>>>>> return 0;
>>>>>> }
>>>>>> @Override
>>>>>> public String toString() {
>>>>>>
>>>>>> return Arrays.toString(value);
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> I wrote matrix write,readfields and toString.
>>>>>>
>>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>>
>>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>>
>>>>>> public class MyReducer extends
>>>>>>
>>>>>>
>>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>
>>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>>> Iterable<MatrixWritable> values, Context context){
>>>>>> for(MatrixWritable c : values){
>>>>>>
>>>>>> System.out.println("print value "+c.toString());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>>> i also needed to print my data asmatrix so i tried
>>>>>>
>>>>>> public String toString() {
>>>>>>
>>>>>> String separator = ", ";
>>>>>> StringBuffer result = new StringBuffer();
>>>>>>
>>>>>> // iterate over the first dimension
>>>>>> for (int i = 0; i < value.length; i++) {
>>>>>> // iterate over the second dimension
>>>>>> for(int j = 0; j < value[i].length; j++){
>>>>>> result.append(value[i][j]);
>>>>>> System.out.print(value[i][j]);
>>>>>> result.append(separator);
>>>>>> }
>>>>>> // remove the last separator
>>>>>> result.setLength(result.length() - separator.length());
>>>>>> // add a line break.
>>>>>> result.append("\n");
>>>>>> }
>>>>>>
>>>>>>
>>>>>> return result.toString();
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Can you share the code?
>>>>>>>
>>>>>>> sent from mobile
>>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>>> Amr Shahin , i refered "
>>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Check this out:
>>>>>>>>>
>>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>>> It shows how to create a customer writable. If you have "hadoop
>>>>>>>>> the
>>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>>> data
>>>>>>>>> types.
>>>>>>>>>
>>>>>>>>> Happy Halloween
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>>
>>>>>>>>> > this is my post from stackoverflow
>>>>>>>>> > but i am not getting any response.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>>> mapper.There are
>>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>>> custom
>>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>>> the matrix
>>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>>> the value
>>>>>>>>> > of
>>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be
>>>>>>>>> a matrix.so
>>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>>> > WritableComparable.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > public class MatrixWritable implements
>>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>>> >
>>>>>>>>> > /**
>>>>>>>>> > * @param args
>>>>>>>>> > */
>>>>>>>>> > private double[][] value;
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable() {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > set(new double[0][0]);
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public void set(double[][] value) {
>>>>>>>>> > this.value = value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > public double[][] getValue() {
>>>>>>>>> > return value;
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>>> > System.out.println("write");
>>>>>>>>> > int row=0;
>>>>>>>>> > int col=0;
>>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>>> > row = value.length;
>>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>>> > col = value[i].length;
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > out.writeInt(row);
>>>>>>>>> > out.writeInt(col);
>>>>>>>>> >
>>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>>> "+row+":"+col);
>>>>>>>>> >
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> >
>>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> > //priting array
>>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>>> > }
>>>>>>>>> > System.out.println("");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>>> > int row = in.readInt();
>>>>>>>>> > int col = in.readInt();
>>>>>>>>> >
>>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int hashCode() {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public boolean equals(Object o) {
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > @Override
>>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>>> > return 0;
>>>>>>>>> > }
>>>>>>>>> > @Override
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > return Arrays.toString(value);
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>>> >
>>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>>> >
>>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>>> with
>>>>>>>>> > emiting only custom value- for checking)
>>>>>>>>> >
>>>>>>>>> > public class MyReducer extends
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>>> >
>>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>>> >
>>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>>> toString()
>>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>>> wrong.and i
>>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>>> >
>>>>>>>>> > public String toString() {
>>>>>>>>> >
>>>>>>>>> > String separator = ", ";
>>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>>> >
>>>>>>>>> > // iterate over the first dimension
>>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>>> > // iterate over the second dimension
>>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>>> > result.append(value[i][j]);
>>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>>> > result.append(separator);
>>>>>>>>> > }
>>>>>>>>> > // remove the last separator
>>>>>>>>> > result.setLength(result.length() -
>>>>>>>>> separator.length());
>>>>>>>>> > // add a line break.
>>>>>>>>> > result.append("\n");
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > return result.toString();
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > }
>>>>>>>>> >
>>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>>> datatype
>>>>>>>>> > CompareTo is neccessary right .
>>>>>>>>> >
>>>>>>>>> > 4.so what should i include in that methods
>>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>>> > what are these methods intended for.
>>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > *Thanks & Regards*
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>>> > *
>>>>>>>>> > *Junior Developer
>>>>>>>>> > *
>>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>>> > *
>>>>>>>>> > *
>>>>>>>>> > Amritapuri.
>>>>>>>>> >
>>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>>> > *
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Thanks & Regards*
>>>>>>>>
>>>>>>>> Unmesha Sreeveni U.B
>>>>>>>>
>>>>>>>> *Junior Developer*
>>>>>>>>
>>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>>
>>>>>>>>
>>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
yes .By editing some of my code..... i am able to get my emitted matrix
VALUE in reducer :) :) :) :).
context.write(......, new MatrixWritable(Eval));
But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
CompareTo() method in my MatrixWritable class.
can anyone suggest me.
On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
> beginners.
>
>
> The second one is right :) .*But you wrote also: I need to emit a 2D
> double array as key and value from mapper.*
> *Means, you work with a k-v-pair *
>
> *KVP<Matrix,Matrix>*
>
> *There Matrix is 2-D matrix of double values.*
>
>
> Yes i need to emit 2 matrices,1 key and the other is value.
>
> ie key -----> A*Atrans--------->after multiplication the result will be a
> 2D array which is declared as double (matrix) lets say the result be Matrix
> "*Ekey*"(double[][] Ekey)
>
> value ------> Atrans*D ---------> after multiplication the result will be
> Matrix "*Eval*" (double[][] Eval).
>
> After tat i need to emit these matrix to reducer for further calculations.
>
> so in mapper
> context.write(*Ekey*,*Eval*);
>
> reducer
> i need to do further calculations with these Ekey nd Eval.
>
>
> so i need to emit context.write(matrix,matrix).....for that i created
> MatrixWritable class.
>
> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
> 2.In reducer i gave iterable why becoz my key and value are matrices.That
> y i gave them as iterable. IS nt that right?????
> If wrong how to give the reducer signature.
>
>
>
>
>
>
>
>
>
>
> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(*Iterable<MatrixWritable>* key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> Usually a key is only one object, not an Iterable.
>>
>> To make things more clear:
>>
>> What is the exact k-v-pair you need in the Reducer?
>>
>> One matrix is the key, and a set of (two matrices together) are used as
>> value in the Reducer? What I understood from your question is
>> KVP<Matrix,Matrix[2]>
>>
>>
>> *But you wrote also:* I need to emit a 2D double array as key and value
>> from mapper.
>> Means, you work with a k-v-pair
>>
>> KVP<Matrix,Matrix>
>>
>> There Matrix is 2-D matrix of double values.
>>
>> I suggest:
>>
>> 1.) Define the MR Data Flow.
>> 2.) Build the custom types.
>> 3.) Test the flow (no computation)
>> 4.) Implement logic / computation
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>
>>> I tried with TwoDArrayWritable too.
>>>
>>> but i tried it by emitting only one value.
>>>
>>> row = E.length;
>>> col = E[0].length;
>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>> // set values in myInnerArray
>>> for (int k1 = 0; k1 < row; k1++) {
>>> for(int j1=0;j1< col;j1++){
>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>
>>> }
>>> array.set(myInnerArray);
>>> context.write(clusterNumber, array);
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> this is also not working for me
>>>
>>> showing NullPointerException
>>>
>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>> java.lang.Exception: java.lang.NullPointerException
>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>> Caused by: java.lang.NullPointerException
>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>> at edu.Mapper.map(Mapper.java:277)
>>>
>>>
>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr Shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Can you share the code?
>>>>>>
>>>>>> sent from mobile
>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>> Amr Shahin , i refered "
>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Check this out:
>>>>>>>>
>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>> data
>>>>>>>> types.
>>>>>>>>
>>>>>>>> Happy Halloween
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>
>>>>>>>> > this is my post from stackoverflow
>>>>>>>> > but i am not getting any response.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>> mapper.There are
>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>> custom
>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>> the matrix
>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>> the value
>>>>>>>> > of
>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>>> matrix.so
>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>> > WritableComparable.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > public class MatrixWritable implements
>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>> >
>>>>>>>> > /**
>>>>>>>> > * @param args
>>>>>>>> > */
>>>>>>>> > private double[][] value;
>>>>>>>> >
>>>>>>>> > public MatrixWritable() {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > set(new double[0][0]);
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public void set(double[][] value) {
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public double[][] getValue() {
>>>>>>>> > return value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>> > System.out.println("write");
>>>>>>>> > int row=0;
>>>>>>>> > int col=0;
>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>> > row = value.length;
>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>> > col = value[i].length;
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > out.writeInt(row);
>>>>>>>> > out.writeInt(col);
>>>>>>>> >
>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>> "+row+":"+col);
>>>>>>>> >
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> >
>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > //priting array
>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>> > }
>>>>>>>> > System.out.println("");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>> > int row = in.readInt();
>>>>>>>> > int col = in.readInt();
>>>>>>>> >
>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int hashCode() {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public boolean equals(Object o) {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>> > return 0;
>>>>>>>> > }
>>>>>>>> > @Override
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > return Arrays.toString(value);
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>> >
>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>> >
>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>> with
>>>>>>>> > emiting only custom value- for checking)
>>>>>>>> >
>>>>>>>> > public class MyReducer extends
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>> >
>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>> >
>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>> toString()
>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>> wrong.and i
>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>> >
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > String separator = ", ";
>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>> >
>>>>>>>> > // iterate over the first dimension
>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>> > // iterate over the second dimension
>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>> > result.append(value[i][j]);
>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>> > result.append(separator);
>>>>>>>> > }
>>>>>>>> > // remove the last separator
>>>>>>>> > result.setLength(result.length() -
>>>>>>>> separator.length());
>>>>>>>> > // add a line break.
>>>>>>>> > result.append("\n");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > return result.toString();
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>> datatype
>>>>>>>> > CompareTo is neccessary right .
>>>>>>>> >
>>>>>>>> > 4.so what should i include in that methods
>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>> > what are these methods intended for.
>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > *Thanks & Regards*
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>> > *
>>>>>>>> > *Junior Developer
>>>>>>>> > *
>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Amritapuri.
>>>>>>>> >
>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>> > *
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Thanks & Regards*
>>>>>>>
>>>>>>> Unmesha Sreeveni U.B
>>>>>>>
>>>>>>> *Junior Developer*
>>>>>>>
>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>
>>>>>>>
>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
yes .By editing some of my code..... i am able to get my emitted matrix
VALUE in reducer :) :) :) :).
context.write(......, new MatrixWritable(Eval));
But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
CompareTo() method in my MatrixWritable class.
can anyone suggest me.
On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
> beginners.
>
>
> The second one is right :) .*But you wrote also: I need to emit a 2D
> double array as key and value from mapper.*
> *Means, you work with a k-v-pair *
>
> *KVP<Matrix,Matrix>*
>
> *There Matrix is 2-D matrix of double values.*
>
>
> Yes i need to emit 2 matrices,1 key and the other is value.
>
> ie key -----> A*Atrans--------->after multiplication the result will be a
> 2D array which is declared as double (matrix) lets say the result be Matrix
> "*Ekey*"(double[][] Ekey)
>
> value ------> Atrans*D ---------> after multiplication the result will be
> Matrix "*Eval*" (double[][] Eval).
>
> After tat i need to emit these matrix to reducer for further calculations.
>
> so in mapper
> context.write(*Ekey*,*Eval*);
>
> reducer
> i need to do further calculations with these Ekey nd Eval.
>
>
> so i need to emit context.write(matrix,matrix).....for that i created
> MatrixWritable class.
>
> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
> 2.In reducer i gave iterable why becoz my key and value are matrices.That
> y i gave them as iterable. IS nt that right?????
> If wrong how to give the reducer signature.
>
>
>
>
>
>
>
>
>
>
> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(*Iterable<MatrixWritable>* key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> Usually a key is only one object, not an Iterable.
>>
>> To make things more clear:
>>
>> What is the exact k-v-pair you need in the Reducer?
>>
>> One matrix is the key, and a set of (two matrices together) are used as
>> value in the Reducer? What I understood from your question is
>> KVP<Matrix,Matrix[2]>
>>
>>
>> *But you wrote also:* I need to emit a 2D double array as key and value
>> from mapper.
>> Means, you work with a k-v-pair
>>
>> KVP<Matrix,Matrix>
>>
>> There Matrix is 2-D matrix of double values.
>>
>> I suggest:
>>
>> 1.) Define the MR Data Flow.
>> 2.) Build the custom types.
>> 3.) Test the flow (no computation)
>> 4.) Implement logic / computation
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>
>>> I tried with TwoDArrayWritable too.
>>>
>>> but i tried it by emitting only one value.
>>>
>>> row = E.length;
>>> col = E[0].length;
>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>> // set values in myInnerArray
>>> for (int k1 = 0; k1 < row; k1++) {
>>> for(int j1=0;j1< col;j1++){
>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>
>>> }
>>> array.set(myInnerArray);
>>> context.write(clusterNumber, array);
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> this is also not working for me
>>>
>>> showing NullPointerException
>>>
>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>> java.lang.Exception: java.lang.NullPointerException
>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>> Caused by: java.lang.NullPointerException
>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>> at edu.Mapper.map(Mapper.java:277)
>>>
>>>
>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr Shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Can you share the code?
>>>>>>
>>>>>> sent from mobile
>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>> Amr Shahin , i refered "
>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Check this out:
>>>>>>>>
>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>> data
>>>>>>>> types.
>>>>>>>>
>>>>>>>> Happy Halloween
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>
>>>>>>>> > this is my post from stackoverflow
>>>>>>>> > but i am not getting any response.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>> mapper.There are
>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>> custom
>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>> the matrix
>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>> the value
>>>>>>>> > of
>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>>> matrix.so
>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>> > WritableComparable.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > public class MatrixWritable implements
>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>> >
>>>>>>>> > /**
>>>>>>>> > * @param args
>>>>>>>> > */
>>>>>>>> > private double[][] value;
>>>>>>>> >
>>>>>>>> > public MatrixWritable() {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > set(new double[0][0]);
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public void set(double[][] value) {
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public double[][] getValue() {
>>>>>>>> > return value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>> > System.out.println("write");
>>>>>>>> > int row=0;
>>>>>>>> > int col=0;
>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>> > row = value.length;
>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>> > col = value[i].length;
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > out.writeInt(row);
>>>>>>>> > out.writeInt(col);
>>>>>>>> >
>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>> "+row+":"+col);
>>>>>>>> >
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> >
>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > //priting array
>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>> > }
>>>>>>>> > System.out.println("");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>> > int row = in.readInt();
>>>>>>>> > int col = in.readInt();
>>>>>>>> >
>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int hashCode() {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public boolean equals(Object o) {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>> > return 0;
>>>>>>>> > }
>>>>>>>> > @Override
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > return Arrays.toString(value);
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>> >
>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>> >
>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>> with
>>>>>>>> > emiting only custom value- for checking)
>>>>>>>> >
>>>>>>>> > public class MyReducer extends
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>> >
>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>> >
>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>> toString()
>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>> wrong.and i
>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>> >
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > String separator = ", ";
>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>> >
>>>>>>>> > // iterate over the first dimension
>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>> > // iterate over the second dimension
>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>> > result.append(value[i][j]);
>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>> > result.append(separator);
>>>>>>>> > }
>>>>>>>> > // remove the last separator
>>>>>>>> > result.setLength(result.length() -
>>>>>>>> separator.length());
>>>>>>>> > // add a line break.
>>>>>>>> > result.append("\n");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > return result.toString();
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>> datatype
>>>>>>>> > CompareTo is neccessary right .
>>>>>>>> >
>>>>>>>> > 4.so what should i include in that methods
>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>> > what are these methods intended for.
>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > *Thanks & Regards*
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>> > *
>>>>>>>> > *Junior Developer
>>>>>>>> > *
>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Amritapuri.
>>>>>>>> >
>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>> > *
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Thanks & Regards*
>>>>>>>
>>>>>>> Unmesha Sreeveni U.B
>>>>>>>
>>>>>>> *Junior Developer*
>>>>>>>
>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>
>>>>>>>
>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
yes .By editing some of my code..... i am able to get my emitted matrix
VALUE in reducer :) :) :) :).
context.write(......, new MatrixWritable(Eval));
But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
CompareTo() method in my MatrixWritable class.
can anyone suggest me.
On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
> beginners.
>
>
> The second one is right :) .*But you wrote also: I need to emit a 2D
> double array as key and value from mapper.*
> *Means, you work with a k-v-pair *
>
> *KVP<Matrix,Matrix>*
>
> *There Matrix is 2-D matrix of double values.*
>
>
> Yes i need to emit 2 matrices,1 key and the other is value.
>
> ie key -----> A*Atrans--------->after multiplication the result will be a
> 2D array which is declared as double (matrix) lets say the result be Matrix
> "*Ekey*"(double[][] Ekey)
>
> value ------> Atrans*D ---------> after multiplication the result will be
> Matrix "*Eval*" (double[][] Eval).
>
> After tat i need to emit these matrix to reducer for further calculations.
>
> so in mapper
> context.write(*Ekey*,*Eval*);
>
> reducer
> i need to do further calculations with these Ekey nd Eval.
>
>
> so i need to emit context.write(matrix,matrix).....for that i created
> MatrixWritable class.
>
> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
> 2.In reducer i gave iterable why becoz my key and value are matrices.That
> y i gave them as iterable. IS nt that right?????
> If wrong how to give the reducer signature.
>
>
>
>
>
>
>
>
>
>
> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(*Iterable<MatrixWritable>* key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> Usually a key is only one object, not an Iterable.
>>
>> To make things more clear:
>>
>> What is the exact k-v-pair you need in the Reducer?
>>
>> One matrix is the key, and a set of (two matrices together) are used as
>> value in the Reducer? What I understood from your question is
>> KVP<Matrix,Matrix[2]>
>>
>>
>> *But you wrote also:* I need to emit a 2D double array as key and value
>> from mapper.
>> Means, you work with a k-v-pair
>>
>> KVP<Matrix,Matrix>
>>
>> There Matrix is 2-D matrix of double values.
>>
>> I suggest:
>>
>> 1.) Define the MR Data Flow.
>> 2.) Build the custom types.
>> 3.) Test the flow (no computation)
>> 4.) Implement logic / computation
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>
>>> I tried with TwoDArrayWritable too.
>>>
>>> but i tried it by emitting only one value.
>>>
>>> row = E.length;
>>> col = E[0].length;
>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>> // set values in myInnerArray
>>> for (int k1 = 0; k1 < row; k1++) {
>>> for(int j1=0;j1< col;j1++){
>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>
>>> }
>>> array.set(myInnerArray);
>>> context.write(clusterNumber, array);
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> this is also not working for me
>>>
>>> showing NullPointerException
>>>
>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>> java.lang.Exception: java.lang.NullPointerException
>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>> Caused by: java.lang.NullPointerException
>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>> at edu.Mapper.map(Mapper.java:277)
>>>
>>>
>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr Shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Can you share the code?
>>>>>>
>>>>>> sent from mobile
>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>> Amr Shahin , i refered "
>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Check this out:
>>>>>>>>
>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>> data
>>>>>>>> types.
>>>>>>>>
>>>>>>>> Happy Halloween
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>
>>>>>>>> > this is my post from stackoverflow
>>>>>>>> > but i am not getting any response.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>> mapper.There are
>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>> custom
>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>> the matrix
>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>> the value
>>>>>>>> > of
>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>>> matrix.so
>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>> > WritableComparable.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > public class MatrixWritable implements
>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>> >
>>>>>>>> > /**
>>>>>>>> > * @param args
>>>>>>>> > */
>>>>>>>> > private double[][] value;
>>>>>>>> >
>>>>>>>> > public MatrixWritable() {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > set(new double[0][0]);
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public void set(double[][] value) {
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public double[][] getValue() {
>>>>>>>> > return value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>> > System.out.println("write");
>>>>>>>> > int row=0;
>>>>>>>> > int col=0;
>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>> > row = value.length;
>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>> > col = value[i].length;
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > out.writeInt(row);
>>>>>>>> > out.writeInt(col);
>>>>>>>> >
>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>> "+row+":"+col);
>>>>>>>> >
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> >
>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > //priting array
>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>> > }
>>>>>>>> > System.out.println("");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>> > int row = in.readInt();
>>>>>>>> > int col = in.readInt();
>>>>>>>> >
>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int hashCode() {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public boolean equals(Object o) {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>> > return 0;
>>>>>>>> > }
>>>>>>>> > @Override
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > return Arrays.toString(value);
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>> >
>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>> >
>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>> with
>>>>>>>> > emiting only custom value- for checking)
>>>>>>>> >
>>>>>>>> > public class MyReducer extends
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>> >
>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>> >
>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>> toString()
>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>> wrong.and i
>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>> >
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > String separator = ", ";
>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>> >
>>>>>>>> > // iterate over the first dimension
>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>> > // iterate over the second dimension
>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>> > result.append(value[i][j]);
>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>> > result.append(separator);
>>>>>>>> > }
>>>>>>>> > // remove the last separator
>>>>>>>> > result.setLength(result.length() -
>>>>>>>> separator.length());
>>>>>>>> > // add a line break.
>>>>>>>> > result.append("\n");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > return result.toString();
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>> datatype
>>>>>>>> > CompareTo is neccessary right .
>>>>>>>> >
>>>>>>>> > 4.so what should i include in that methods
>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>> > what are these methods intended for.
>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > *Thanks & Regards*
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>> > *
>>>>>>>> > *Junior Developer
>>>>>>>> > *
>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Amritapuri.
>>>>>>>> >
>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>> > *
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Thanks & Regards*
>>>>>>>
>>>>>>> Unmesha Sreeveni U.B
>>>>>>>
>>>>>>> *Junior Developer*
>>>>>>>
>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>
>>>>>>>
>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
yes .By editing some of my code..... i am able to get my emitted matrix
VALUE in reducer :) :) :) :).
context.write(......, new MatrixWritable(Eval));
But i am confused , HOW TO EMIT A KEY too from mapper ..what will be my
CompareTo() method in my MatrixWritable class.
can anyone suggest me.
On Sun, Nov 3, 2013 at 11:33 PM, unmesha sreeveni <un...@gmail.com>wrote:
> Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
> beginners.
>
>
> The second one is right :) .*But you wrote also: I need to emit a 2D
> double array as key and value from mapper.*
> *Means, you work with a k-v-pair *
>
> *KVP<Matrix,Matrix>*
>
> *There Matrix is 2-D matrix of double values.*
>
>
> Yes i need to emit 2 matrices,1 key and the other is value.
>
> ie key -----> A*Atrans--------->after multiplication the result will be a
> 2D array which is declared as double (matrix) lets say the result be Matrix
> "*Ekey*"(double[][] Ekey)
>
> value ------> Atrans*D ---------> after multiplication the result will be
> Matrix "*Eval*" (double[][] Eval).
>
> After tat i need to emit these matrix to reducer for further calculations.
>
> so in mapper
> context.write(*Ekey*,*Eval*);
>
> reducer
> i need to do further calculations with these Ekey nd Eval.
>
>
> so i need to emit context.write(matrix,matrix).....for that i created
> MatrixWritable class.
>
> 1.Is that the correct way or i can directly go for TwoDArrayWritable?
> 2.In reducer i gave iterable why becoz my key and value are matrices.That
> y i gave them as iterable. IS nt that right?????
> If wrong how to give the reducer signature.
>
>
>
>
>
>
>
>
>
>
> On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com>wrote:
>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(*Iterable<MatrixWritable>* key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> Usually a key is only one object, not an Iterable.
>>
>> To make things more clear:
>>
>> What is the exact k-v-pair you need in the Reducer?
>>
>> One matrix is the key, and a set of (two matrices together) are used as
>> value in the Reducer? What I understood from your question is
>> KVP<Matrix,Matrix[2]>
>>
>>
>> *But you wrote also:* I need to emit a 2D double array as key and value
>> from mapper.
>> Means, you work with a k-v-pair
>>
>> KVP<Matrix,Matrix>
>>
>> There Matrix is 2-D matrix of double values.
>>
>> I suggest:
>>
>> 1.) Define the MR Data Flow.
>> 2.) Build the custom types.
>> 3.) Test the flow (no computation)
>> 4.) Implement logic / computation
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>>
>>> I tried with TwoDArrayWritable too.
>>>
>>> but i tried it by emitting only one value.
>>>
>>> row = E.length;
>>> col = E[0].length;
>>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>>> // set values in myInnerArray
>>> for (int k1 = 0; k1 < row; k1++) {
>>> for(int j1=0;j1< col;j1++){
>>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>>
>>> }
>>> array.set(myInnerArray);
>>> context.write(clusterNumber, array);
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> this is also not working for me
>>>
>>> showing NullPointerException
>>>
>>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>>> java.lang.Exception: java.lang.NullPointerException
>>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>>> Caused by: java.lang.NullPointerException
>>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>>> at edu.Mapper.map(Mapper.java:277)
>>>
>>>
>>> Mapper.java:277 : context.write(clusterNumber, array);
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr Shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>>> > wrote:
>>>>
>>>>> @Amr shahin
>>>>> this is my post from stackoverflow
>>>>> but i am not getting any response.
>>>>>
>>>>>
>>>>> I need to emit a 2D double array as key and value from mapper.There
>>>>> are questions posted in stackoverflow. But they are not answered.
>>>>> we have to create a custom datatype.but how?I am new to these custom
>>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>>> be corresponding to the key itself.I think for that we need to use
>>>>> WritableComparable.
>>>>>
>>>>>
>>>>>
>>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>>
>>>>> /**
>>>>> * @param args
>>>>> */
>>>>> private double[][] value;
>>>>>
>>>>> public MatrixWritable() {
>>>>> // TODO Auto-generated constructor stub
>>>>> set(new double[0][0]);
>>>>> }
>>>>>
>>>>> public MatrixWritable(double[][] value) {
>>>>> // TODO Auto-generated constructor stub
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public void set(double[][] value) {
>>>>> this.value = value;
>>>>> }
>>>>>
>>>>> public double[][] getValue() {
>>>>> return value;
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void write(DataOutput out) throws IOException {
>>>>> System.out.println("write");
>>>>> int row=0;
>>>>> int col=0;
>>>>> for(int i=0; i<value.length;i++){
>>>>> row = value.length;
>>>>> for(int j=0; j<value[i].length; j++){
>>>>> col = value[i].length;
>>>>> }
>>>>> }
>>>>> out.writeInt(row);
>>>>> out.writeInt(col);
>>>>>
>>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>>
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>>
>>>>> out.writeDouble(value[i][j]);
>>>>> }
>>>>> }
>>>>> //priting array
>>>>> for(int vali =0;vali< value.length ;vali ++){
>>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>>> System.out.print(value[vali][valj]+ "\t");
>>>>> }
>>>>> System.out.println("");
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public void readFields(DataInput in) throws IOException {
>>>>> int row = in.readInt();
>>>>> int col = in.readInt();
>>>>>
>>>>> double[][] value = new double[row][col];
>>>>> for(int i=0;i<row ; i++){
>>>>> for(int j= 0 ; j< col;j++){
>>>>> value[i][j] = in.readDouble();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public int hashCode() {
>>>>>
>>>>> }
>>>>>
>>>>> @Override
>>>>> public boolean equals(Object o) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> @Override
>>>>> public int compareTo(MatrixWritable o) {
>>>>> // TODO Auto-generated method stub
>>>>> return 0;
>>>>> }
>>>>> @Override
>>>>> public String toString() {
>>>>>
>>>>> return Arrays.toString(value);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> I wrote matrix write,readfields and toString.
>>>>>
>>>>> 1.But my toString is not returning anything. why is it so?
>>>>>
>>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>>
>>>>> public class MyReducer extends
>>>>>
>>>>>
>>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>
>>>>> public void reduce(Iterable<MatrixWritable> key,
>>>>> Iterable<MatrixWritable> values, Context context){
>>>>> for(MatrixWritable c : values){
>>>>>
>>>>> System.out.println("print value "+c.toString());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>>> i also needed to print my data asmatrix so i tried
>>>>>
>>>>> public String toString() {
>>>>>
>>>>> String separator = ", ";
>>>>> StringBuffer result = new StringBuffer();
>>>>>
>>>>> // iterate over the first dimension
>>>>> for (int i = 0; i < value.length; i++) {
>>>>> // iterate over the second dimension
>>>>> for(int j = 0; j < value[i].length; j++){
>>>>> result.append(value[i][j]);
>>>>> System.out.print(value[i][j]);
>>>>> result.append(separator);
>>>>> }
>>>>> // remove the last separator
>>>>> result.setLength(result.length() - separator.length());
>>>>> // add a line break.
>>>>> result.append("\n");
>>>>> }
>>>>>
>>>>>
>>>>> return result.toString();
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Can you share the code?
>>>>>>
>>>>>> sent from mobile
>>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>>> Amr Shahin , i refered "
>>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Check this out:
>>>>>>>>
>>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>>> data
>>>>>>>> types.
>>>>>>>>
>>>>>>>> Happy Halloween
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>>
>>>>>>>> > this is my post from stackoverflow
>>>>>>>> > but i am not getting any response.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>>> mapper.There are
>>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>>> custom
>>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>>> the matrix
>>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>>> the value
>>>>>>>> > of
>>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>>> matrix.so
>>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>>> > WritableComparable.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > public class MatrixWritable implements
>>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>>> >
>>>>>>>> > /**
>>>>>>>> > * @param args
>>>>>>>> > */
>>>>>>>> > private double[][] value;
>>>>>>>> >
>>>>>>>> > public MatrixWritable() {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > set(new double[0][0]);
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public void set(double[][] value) {
>>>>>>>> > this.value = value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > public double[][] getValue() {
>>>>>>>> > return value;
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>>> > System.out.println("write");
>>>>>>>> > int row=0;
>>>>>>>> > int col=0;
>>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>>> > row = value.length;
>>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>>> > col = value[i].length;
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > out.writeInt(row);
>>>>>>>> > out.writeInt(col);
>>>>>>>> >
>>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>>> "+row+":"+col);
>>>>>>>> >
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> >
>>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> > //priting array
>>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>>> > }
>>>>>>>> > System.out.println("");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>>> > int row = in.readInt();
>>>>>>>> > int col = in.readInt();
>>>>>>>> >
>>>>>>>> > double[][] value = new double[row][col];
>>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>>> > value[i][j] = in.readDouble();
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int hashCode() {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public boolean equals(Object o) {
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > @Override
>>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>>> > // TODO Auto-generated method stub
>>>>>>>> > return 0;
>>>>>>>> > }
>>>>>>>> > @Override
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > return Arrays.toString(value);
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>>> >
>>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>>> >
>>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>>> with
>>>>>>>> > emiting only custom value- for checking)
>>>>>>>> >
>>>>>>>> > public class MyReducer extends
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>>> >
>>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>>> > for(MatrixWritable c : values){
>>>>>>>> >
>>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>>> toString()
>>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>>> wrong.and i
>>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>>> >
>>>>>>>> > public String toString() {
>>>>>>>> >
>>>>>>>> > String separator = ", ";
>>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>>> >
>>>>>>>> > // iterate over the first dimension
>>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>>> > // iterate over the second dimension
>>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>>> > result.append(value[i][j]);
>>>>>>>> > System.out.print(value[i][j]);
>>>>>>>> > result.append(separator);
>>>>>>>> > }
>>>>>>>> > // remove the last separator
>>>>>>>> > result.setLength(result.length() -
>>>>>>>> separator.length());
>>>>>>>> > // add a line break.
>>>>>>>> > result.append("\n");
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > return result.toString();
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > }
>>>>>>>> >
>>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>>> datatype
>>>>>>>> > CompareTo is neccessary right .
>>>>>>>> >
>>>>>>>> > 4.so what should i include in that methods
>>>>>>>> CompareTo,hashcode,equals and
>>>>>>>> > what are these methods intended for.
>>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > *Thanks & Regards*
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>>> > *
>>>>>>>> > *Junior Developer
>>>>>>>> > *
>>>>>>>> > *Amrita Center For Cyber Security
>>>>>>>> > *
>>>>>>>> > *
>>>>>>>> > Amritapuri.
>>>>>>>> >
>>>>>>>> > www.amrita.edu/cyber/
>>>>>>>> > *
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Thanks & Regards*
>>>>>>>
>>>>>>> Unmesha Sreeveni U.B
>>>>>>>
>>>>>>> *Junior Developer*
>>>>>>>
>>>>>>> *Amrita Center For Cyber Security *
>>>>>>>
>>>>>>>
>>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
beginners.
The second one is right :) .*But you wrote also: I need to emit a 2D double
array as key and value from mapper.*
*Means, you work with a k-v-pair *
*KVP<Matrix,Matrix>*
*There Matrix is 2-D matrix of double values.*
Yes i need to emit 2 matrices,1 key and the other is value.
ie key -----> A*Atrans--------->after multiplication the result will be a
2D array which is declared as double (matrix) lets say the result be Matrix
"*Ekey*"(double[][] Ekey)
value ------> Atrans*D ---------> after multiplication the result will be
Matrix "*Eval*" (double[][] Eval).
After tat i need to emit these matrix to reducer for further calculations.
so in mapper
context.write(*Ekey*,*Eval*);
reducer
i need to do further calculations with these Ekey nd Eval.
so i need to emit context.write(matrix,matrix).....for that i created
MatrixWritable class.
1.Is that the correct way or i can directly go for TwoDArrayWritable?
2.In reducer i gave iterable why becoz my key and value are matrices.That y
i gave them as iterable. IS nt that right?????
If wrong how to give the reducer signature.
On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com> wrote:
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(*Iterable<MatrixWritable>* key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> Usually a key is only one object, not an Iterable.
>
> To make things more clear:
>
> What is the exact k-v-pair you need in the Reducer?
>
> One matrix is the key, and a set of (two matrices together) are used as
> value in the Reducer? What I understood from your question is
> KVP<Matrix,Matrix[2]>
>
>
> *But you wrote also:* I need to emit a 2D double array as key and value
> from mapper.
> Means, you work with a k-v-pair
>
> KVP<Matrix,Matrix>
>
> There Matrix is 2-D matrix of double values.
>
> I suggest:
>
> 1.) Define the MR Data Flow.
> 2.) Build the custom types.
> 3.) Test the flow (no computation)
> 4.) Implement logic / computation
>
>
>
>
>
>
>
>
> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>
>> I tried with TwoDArrayWritable too.
>>
>> but i tried it by emitting only one value.
>>
>> row = E.length;
>> col = E[0].length;
>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>> // set values in myInnerArray
>> for (int k1 = 0; k1 < row; k1++) {
>> for(int j1=0;j1< col;j1++){
>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>
>> }
>> array.set(myInnerArray);
>> context.write(clusterNumber, array);
>>
>>
>>
>>
>>
>>
>>
>> this is also not working for me
>>
>> showing NullPointerException
>>
>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>> java.lang.Exception: java.lang.NullPointerException
>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>> at edu.Mapper.map(Mapper.java:277)
>>
>>
>> Mapper.java:277 : context.write(clusterNumber, array);
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr Shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>>
>>>>> Can you share the code?
>>>>>
>>>>> sent from mobile
>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>> Amr Shahin , i refered "
>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Check this out:
>>>>>>>
>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>> data
>>>>>>> types.
>>>>>>>
>>>>>>> Happy Halloween
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>
>>>>>>> > this is my post from stackoverflow
>>>>>>> > but i am not getting any response.
>>>>>>> >
>>>>>>> >
>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>> mapper.There are
>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>> custom
>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>> the matrix
>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>> the value
>>>>>>> > of
>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>> matrix.so
>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>> > WritableComparable.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > public class MatrixWritable implements
>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>> >
>>>>>>> > /**
>>>>>>> > * @param args
>>>>>>> > */
>>>>>>> > private double[][] value;
>>>>>>> >
>>>>>>> > public MatrixWritable() {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > set(new double[0][0]);
>>>>>>> > }
>>>>>>> >
>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public void set(double[][] value) {
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public double[][] getValue() {
>>>>>>> > return value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>> > System.out.println("write");
>>>>>>> > int row=0;
>>>>>>> > int col=0;
>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>> > row = value.length;
>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>> > col = value[i].length;
>>>>>>> > }
>>>>>>> > }
>>>>>>> > out.writeInt(row);
>>>>>>> > out.writeInt(col);
>>>>>>> >
>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>> "+row+":"+col);
>>>>>>> >
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> >
>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>> > }
>>>>>>> > }
>>>>>>> > //priting array
>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>> > }
>>>>>>> > System.out.println("");
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>> > int row = in.readInt();
>>>>>>> > int col = in.readInt();
>>>>>>> >
>>>>>>> > double[][] value = new double[row][col];
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> > value[i][j] = in.readDouble();
>>>>>>> >
>>>>>>> > }
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int hashCode() {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public boolean equals(Object o) {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>> > // TODO Auto-generated method stub
>>>>>>> > return 0;
>>>>>>> > }
>>>>>>> > @Override
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > return Arrays.toString(value);
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>> >
>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>> >
>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>> with
>>>>>>> > emiting only custom value- for checking)
>>>>>>> >
>>>>>>> > public class MyReducer extends
>>>>>>> >
>>>>>>> >
>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>> >
>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>> > for(MatrixWritable c : values){
>>>>>>> >
>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>> toString()
>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>> wrong.and i
>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>> >
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > String separator = ", ";
>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>> >
>>>>>>> > // iterate over the first dimension
>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>> > // iterate over the second dimension
>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>> > result.append(value[i][j]);
>>>>>>> > System.out.print(value[i][j]);
>>>>>>> > result.append(separator);
>>>>>>> > }
>>>>>>> > // remove the last separator
>>>>>>> > result.setLength(result.length() - separator.length());
>>>>>>> > // add a line break.
>>>>>>> > result.append("\n");
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > return result.toString();
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>> datatype
>>>>>>> > CompareTo is neccessary right .
>>>>>>> >
>>>>>>> > 4.so what should i include in that methods
>>>>>>> CompareTo,hashcode,equals and
>>>>>>> > what are these methods intended for.
>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>> >
>>>>>>> > --
>>>>>>> > *Thanks & Regards*
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>> > *
>>>>>>> > *Junior Developer
>>>>>>> > *
>>>>>>> > *Amrita Center For Cyber Security
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Amritapuri.
>>>>>>> >
>>>>>>> > www.amrita.edu/cyber/
>>>>>>> > *
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
beginners.
The second one is right :) .*But you wrote also: I need to emit a 2D double
array as key and value from mapper.*
*Means, you work with a k-v-pair *
*KVP<Matrix,Matrix>*
*There Matrix is 2-D matrix of double values.*
Yes i need to emit 2 matrices,1 key and the other is value.
ie key -----> A*Atrans--------->after multiplication the result will be a
2D array which is declared as double (matrix) lets say the result be Matrix
"*Ekey*"(double[][] Ekey)
value ------> Atrans*D ---------> after multiplication the result will be
Matrix "*Eval*" (double[][] Eval).
After tat i need to emit these matrix to reducer for further calculations.
so in mapper
context.write(*Ekey*,*Eval*);
reducer
i need to do further calculations with these Ekey nd Eval.
so i need to emit context.write(matrix,matrix).....for that i created
MatrixWritable class.
1.Is that the correct way or i can directly go for TwoDArrayWritable?
2.In reducer i gave iterable why becoz my key and value are matrices.That y
i gave them as iterable. IS nt that right?????
If wrong how to give the reducer signature.
On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com> wrote:
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(*Iterable<MatrixWritable>* key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> Usually a key is only one object, not an Iterable.
>
> To make things more clear:
>
> What is the exact k-v-pair you need in the Reducer?
>
> One matrix is the key, and a set of (two matrices together) are used as
> value in the Reducer? What I understood from your question is
> KVP<Matrix,Matrix[2]>
>
>
> *But you wrote also:* I need to emit a 2D double array as key and value
> from mapper.
> Means, you work with a k-v-pair
>
> KVP<Matrix,Matrix>
>
> There Matrix is 2-D matrix of double values.
>
> I suggest:
>
> 1.) Define the MR Data Flow.
> 2.) Build the custom types.
> 3.) Test the flow (no computation)
> 4.) Implement logic / computation
>
>
>
>
>
>
>
>
> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>
>> I tried with TwoDArrayWritable too.
>>
>> but i tried it by emitting only one value.
>>
>> row = E.length;
>> col = E[0].length;
>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>> // set values in myInnerArray
>> for (int k1 = 0; k1 < row; k1++) {
>> for(int j1=0;j1< col;j1++){
>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>
>> }
>> array.set(myInnerArray);
>> context.write(clusterNumber, array);
>>
>>
>>
>>
>>
>>
>>
>> this is also not working for me
>>
>> showing NullPointerException
>>
>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>> java.lang.Exception: java.lang.NullPointerException
>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>> at edu.Mapper.map(Mapper.java:277)
>>
>>
>> Mapper.java:277 : context.write(clusterNumber, array);
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr Shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>>
>>>>> Can you share the code?
>>>>>
>>>>> sent from mobile
>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>> Amr Shahin , i refered "
>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Check this out:
>>>>>>>
>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>> data
>>>>>>> types.
>>>>>>>
>>>>>>> Happy Halloween
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>
>>>>>>> > this is my post from stackoverflow
>>>>>>> > but i am not getting any response.
>>>>>>> >
>>>>>>> >
>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>> mapper.There are
>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>> custom
>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>> the matrix
>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>> the value
>>>>>>> > of
>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>> matrix.so
>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>> > WritableComparable.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > public class MatrixWritable implements
>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>> >
>>>>>>> > /**
>>>>>>> > * @param args
>>>>>>> > */
>>>>>>> > private double[][] value;
>>>>>>> >
>>>>>>> > public MatrixWritable() {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > set(new double[0][0]);
>>>>>>> > }
>>>>>>> >
>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public void set(double[][] value) {
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public double[][] getValue() {
>>>>>>> > return value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>> > System.out.println("write");
>>>>>>> > int row=0;
>>>>>>> > int col=0;
>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>> > row = value.length;
>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>> > col = value[i].length;
>>>>>>> > }
>>>>>>> > }
>>>>>>> > out.writeInt(row);
>>>>>>> > out.writeInt(col);
>>>>>>> >
>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>> "+row+":"+col);
>>>>>>> >
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> >
>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>> > }
>>>>>>> > }
>>>>>>> > //priting array
>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>> > }
>>>>>>> > System.out.println("");
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>> > int row = in.readInt();
>>>>>>> > int col = in.readInt();
>>>>>>> >
>>>>>>> > double[][] value = new double[row][col];
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> > value[i][j] = in.readDouble();
>>>>>>> >
>>>>>>> > }
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int hashCode() {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public boolean equals(Object o) {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>> > // TODO Auto-generated method stub
>>>>>>> > return 0;
>>>>>>> > }
>>>>>>> > @Override
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > return Arrays.toString(value);
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>> >
>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>> >
>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>> with
>>>>>>> > emiting only custom value- for checking)
>>>>>>> >
>>>>>>> > public class MyReducer extends
>>>>>>> >
>>>>>>> >
>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>> >
>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>> > for(MatrixWritable c : values){
>>>>>>> >
>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>> toString()
>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>> wrong.and i
>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>> >
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > String separator = ", ";
>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>> >
>>>>>>> > // iterate over the first dimension
>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>> > // iterate over the second dimension
>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>> > result.append(value[i][j]);
>>>>>>> > System.out.print(value[i][j]);
>>>>>>> > result.append(separator);
>>>>>>> > }
>>>>>>> > // remove the last separator
>>>>>>> > result.setLength(result.length() - separator.length());
>>>>>>> > // add a line break.
>>>>>>> > result.append("\n");
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > return result.toString();
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>> datatype
>>>>>>> > CompareTo is neccessary right .
>>>>>>> >
>>>>>>> > 4.so what should i include in that methods
>>>>>>> CompareTo,hashcode,equals and
>>>>>>> > what are these methods intended for.
>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>> >
>>>>>>> > --
>>>>>>> > *Thanks & Regards*
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>> > *
>>>>>>> > *Junior Developer
>>>>>>> > *
>>>>>>> > *Amrita Center For Cyber Security
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Amritapuri.
>>>>>>> >
>>>>>>> > www.amrita.edu/cyber/
>>>>>>> > *
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
beginners.
The second one is right :) .*But you wrote also: I need to emit a 2D double
array as key and value from mapper.*
*Means, you work with a k-v-pair *
*KVP<Matrix,Matrix>*
*There Matrix is 2-D matrix of double values.*
Yes i need to emit 2 matrices,1 key and the other is value.
ie key -----> A*Atrans--------->after multiplication the result will be a
2D array which is declared as double (matrix) lets say the result be Matrix
"*Ekey*"(double[][] Ekey)
value ------> Atrans*D ---------> after multiplication the result will be
Matrix "*Eval*" (double[][] Eval).
After tat i need to emit these matrix to reducer for further calculations.
so in mapper
context.write(*Ekey*,*Eval*);
reducer
i need to do further calculations with these Ekey nd Eval.
so i need to emit context.write(matrix,matrix).....for that i created
MatrixWritable class.
1.Is that the correct way or i can directly go for TwoDArrayWritable?
2.In reducer i gave iterable why becoz my key and value are matrices.That y
i gave them as iterable. IS nt that right?????
If wrong how to give the reducer signature.
On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com> wrote:
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(*Iterable<MatrixWritable>* key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> Usually a key is only one object, not an Iterable.
>
> To make things more clear:
>
> What is the exact k-v-pair you need in the Reducer?
>
> One matrix is the key, and a set of (two matrices together) are used as
> value in the Reducer? What I understood from your question is
> KVP<Matrix,Matrix[2]>
>
>
> *But you wrote also:* I need to emit a 2D double array as key and value
> from mapper.
> Means, you work with a k-v-pair
>
> KVP<Matrix,Matrix>
>
> There Matrix is 2-D matrix of double values.
>
> I suggest:
>
> 1.) Define the MR Data Flow.
> 2.) Build the custom types.
> 3.) Test the flow (no computation)
> 4.) Implement logic / computation
>
>
>
>
>
>
>
>
> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>
>> I tried with TwoDArrayWritable too.
>>
>> but i tried it by emitting only one value.
>>
>> row = E.length;
>> col = E[0].length;
>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>> // set values in myInnerArray
>> for (int k1 = 0; k1 < row; k1++) {
>> for(int j1=0;j1< col;j1++){
>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>
>> }
>> array.set(myInnerArray);
>> context.write(clusterNumber, array);
>>
>>
>>
>>
>>
>>
>>
>> this is also not working for me
>>
>> showing NullPointerException
>>
>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>> java.lang.Exception: java.lang.NullPointerException
>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>> at edu.Mapper.map(Mapper.java:277)
>>
>>
>> Mapper.java:277 : context.write(clusterNumber, array);
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr Shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>>
>>>>> Can you share the code?
>>>>>
>>>>> sent from mobile
>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>> Amr Shahin , i refered "
>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Check this out:
>>>>>>>
>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>> data
>>>>>>> types.
>>>>>>>
>>>>>>> Happy Halloween
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>
>>>>>>> > this is my post from stackoverflow
>>>>>>> > but i am not getting any response.
>>>>>>> >
>>>>>>> >
>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>> mapper.There are
>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>> custom
>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>> the matrix
>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>> the value
>>>>>>> > of
>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>> matrix.so
>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>> > WritableComparable.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > public class MatrixWritable implements
>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>> >
>>>>>>> > /**
>>>>>>> > * @param args
>>>>>>> > */
>>>>>>> > private double[][] value;
>>>>>>> >
>>>>>>> > public MatrixWritable() {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > set(new double[0][0]);
>>>>>>> > }
>>>>>>> >
>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public void set(double[][] value) {
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public double[][] getValue() {
>>>>>>> > return value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>> > System.out.println("write");
>>>>>>> > int row=0;
>>>>>>> > int col=0;
>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>> > row = value.length;
>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>> > col = value[i].length;
>>>>>>> > }
>>>>>>> > }
>>>>>>> > out.writeInt(row);
>>>>>>> > out.writeInt(col);
>>>>>>> >
>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>> "+row+":"+col);
>>>>>>> >
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> >
>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>> > }
>>>>>>> > }
>>>>>>> > //priting array
>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>> > }
>>>>>>> > System.out.println("");
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>> > int row = in.readInt();
>>>>>>> > int col = in.readInt();
>>>>>>> >
>>>>>>> > double[][] value = new double[row][col];
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> > value[i][j] = in.readDouble();
>>>>>>> >
>>>>>>> > }
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int hashCode() {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public boolean equals(Object o) {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>> > // TODO Auto-generated method stub
>>>>>>> > return 0;
>>>>>>> > }
>>>>>>> > @Override
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > return Arrays.toString(value);
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>> >
>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>> >
>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>> with
>>>>>>> > emiting only custom value- for checking)
>>>>>>> >
>>>>>>> > public class MyReducer extends
>>>>>>> >
>>>>>>> >
>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>> >
>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>> > for(MatrixWritable c : values){
>>>>>>> >
>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>> toString()
>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>> wrong.and i
>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>> >
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > String separator = ", ";
>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>> >
>>>>>>> > // iterate over the first dimension
>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>> > // iterate over the second dimension
>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>> > result.append(value[i][j]);
>>>>>>> > System.out.print(value[i][j]);
>>>>>>> > result.append(separator);
>>>>>>> > }
>>>>>>> > // remove the last separator
>>>>>>> > result.setLength(result.length() - separator.length());
>>>>>>> > // add a line break.
>>>>>>> > result.append("\n");
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > return result.toString();
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>> datatype
>>>>>>> > CompareTo is neccessary right .
>>>>>>> >
>>>>>>> > 4.so what should i include in that methods
>>>>>>> CompareTo,hashcode,equals and
>>>>>>> > what are these methods intended for.
>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>> >
>>>>>>> > --
>>>>>>> > *Thanks & Regards*
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>> > *
>>>>>>> > *Junior Developer
>>>>>>> > *
>>>>>>> > *Amrita Center For Cyber Security
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Amritapuri.
>>>>>>> >
>>>>>>> > www.amrita.edu/cyber/
>>>>>>> > *
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks for ur reply..Mirko Kampf. And the suggestion was really good for
beginners.
The second one is right :) .*But you wrote also: I need to emit a 2D double
array as key and value from mapper.*
*Means, you work with a k-v-pair *
*KVP<Matrix,Matrix>*
*There Matrix is 2-D matrix of double values.*
Yes i need to emit 2 matrices,1 key and the other is value.
ie key -----> A*Atrans--------->after multiplication the result will be a
2D array which is declared as double (matrix) lets say the result be Matrix
"*Ekey*"(double[][] Ekey)
value ------> Atrans*D ---------> after multiplication the result will be
Matrix "*Eval*" (double[][] Eval).
After tat i need to emit these matrix to reducer for further calculations.
so in mapper
context.write(*Ekey*,*Eval*);
reducer
i need to do further calculations with these Ekey nd Eval.
so i need to emit context.write(matrix,matrix).....for that i created
MatrixWritable class.
1.Is that the correct way or i can directly go for TwoDArrayWritable?
2.In reducer i gave iterable why becoz my key and value are matrices.That y
i gave them as iterable. IS nt that right?????
If wrong how to give the reducer signature.
On Sun, Nov 3, 2013 at 5:44 PM, Mirko Kämpf <mi...@gmail.com> wrote:
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(*Iterable<MatrixWritable>* key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> Usually a key is only one object, not an Iterable.
>
> To make things more clear:
>
> What is the exact k-v-pair you need in the Reducer?
>
> One matrix is the key, and a set of (two matrices together) are used as
> value in the Reducer? What I understood from your question is
> KVP<Matrix,Matrix[2]>
>
>
> *But you wrote also:* I need to emit a 2D double array as key and value
> from mapper.
> Means, you work with a k-v-pair
>
> KVP<Matrix,Matrix>
>
> There Matrix is 2-D matrix of double values.
>
> I suggest:
>
> 1.) Define the MR Data Flow.
> 2.) Build the custom types.
> 3.) Test the flow (no computation)
> 4.) Implement logic / computation
>
>
>
>
>
>
>
>
> 2013/11/3 unmesha sreeveni <un...@gmail.com>
>
>> I tried with TwoDArrayWritable too.
>>
>> but i tried it by emitting only one value.
>>
>> row = E.length;
>> col = E[0].length;
>> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
>> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
>> // set values in myInnerArray
>> for (int k1 = 0; k1 < row; k1++) {
>> for(int j1=0;j1< col;j1++){
>> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>>
>> }
>> array.set(myInnerArray);
>> context.write(clusterNumber, array);
>>
>>
>>
>>
>>
>>
>>
>> this is also not working for me
>>
>> showing NullPointerException
>>
>> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
>> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
>> java.lang.Exception: java.lang.NullPointerException
>> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
>> Caused by: java.lang.NullPointerException
>> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
>> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
>> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
>> at edu.Mapper.map(Mapper.java:277)
>>
>>
>> Mapper.java:277 : context.write(clusterNumber, array);
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr Shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>>
>>>> @Amr shahin
>>>> this is my post from stackoverflow
>>>> but i am not getting any response.
>>>>
>>>>
>>>> I need to emit a 2D double array as key and value from mapper.There are
>>>> questions posted in stackoverflow. But they are not answered.
>>>> we have to create a custom datatype.but how?I am new to these custom
>>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>>> multiplication in a given dataset.and after that i need to emit the value of
>>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so how to emit these matrices from mapper.And the value should
>>>> be corresponding to the key itself.I think for that we need to use
>>>> WritableComparable.
>>>>
>>>>
>>>>
>>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>>
>>>> /**
>>>> * @param args
>>>> */
>>>> private double[][] value;
>>>>
>>>> public MatrixWritable() {
>>>> // TODO Auto-generated constructor stub
>>>> set(new double[0][0]);
>>>> }
>>>>
>>>> public MatrixWritable(double[][] value) {
>>>> // TODO Auto-generated constructor stub
>>>> this.value = value;
>>>> }
>>>>
>>>> public void set(double[][] value) {
>>>> this.value = value;
>>>> }
>>>>
>>>> public double[][] getValue() {
>>>> return value;
>>>> }
>>>>
>>>> @Override
>>>> public void write(DataOutput out) throws IOException {
>>>> System.out.println("write");
>>>> int row=0;
>>>> int col=0;
>>>> for(int i=0; i<value.length;i++){
>>>> row = value.length;
>>>> for(int j=0; j<value[i].length; j++){
>>>> col = value[i].length;
>>>> }
>>>> }
>>>> out.writeInt(row);
>>>> out.writeInt(col);
>>>>
>>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>>
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>>
>>>> out.writeDouble(value[i][j]);
>>>> }
>>>> }
>>>> //priting array
>>>> for(int vali =0;vali< value.length ;vali ++){
>>>> for(int valj = 0;valj <value[0].length;valj++){
>>>> System.out.print(value[vali][valj]+ "\t");
>>>> }
>>>> System.out.println("");
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public void readFields(DataInput in) throws IOException {
>>>> int row = in.readInt();
>>>> int col = in.readInt();
>>>>
>>>> double[][] value = new double[row][col];
>>>> for(int i=0;i<row ; i++){
>>>> for(int j= 0 ; j< col;j++){
>>>> value[i][j] = in.readDouble();
>>>>
>>>> }
>>>> }
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public int hashCode() {
>>>>
>>>> }
>>>>
>>>> @Override
>>>> public boolean equals(Object o) {
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> public int compareTo(MatrixWritable o) {
>>>> // TODO Auto-generated method stub
>>>> return 0;
>>>> }
>>>> @Override
>>>> public String toString() {
>>>>
>>>> return Arrays.toString(value);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> I wrote matrix write,readfields and toString.
>>>>
>>>> 1.But my toString is not returning anything. why is it so?
>>>>
>>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>>
>>>> public class MyReducer extends
>>>>
>>>>
>>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>
>>>> public void reduce(Iterable<MatrixWritable> key,
>>>> Iterable<MatrixWritable> values, Context context){
>>>> for(MatrixWritable c : values){
>>>>
>>>> System.out.println("print value "+c.toString());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>>> i also needed to print my data asmatrix so i tried
>>>>
>>>> public String toString() {
>>>>
>>>> String separator = ", ";
>>>> StringBuffer result = new StringBuffer();
>>>>
>>>> // iterate over the first dimension
>>>> for (int i = 0; i < value.length; i++) {
>>>> // iterate over the second dimension
>>>> for(int j = 0; j < value[i].length; j++){
>>>> result.append(value[i][j]);
>>>> System.out.print(value[i][j]);
>>>> result.append(separator);
>>>> }
>>>> // remove the last separator
>>>> result.setLength(result.length() - separator.length());
>>>> // add a line break.
>>>> result.append("\n");
>>>> }
>>>>
>>>>
>>>> return result.toString();
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>>
>>>>> Can you share the code?
>>>>>
>>>>> sent from mobile
>>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> thanks Steve Loughran and Amr Shahin
>>>>>> Amr Shahin , i refered "
>>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>>> the same thing only. but my toString is not returning anything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>>
>>>>>>> Check this out:
>>>>>>>
>>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>>> definitive guide" there is a really good explanation about custom
>>>>>>> data
>>>>>>> types.
>>>>>>>
>>>>>>> Happy Halloween
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>>
>>>>>>> > this is my post from stackoverflow
>>>>>>> > but i am not getting any response.
>>>>>>> >
>>>>>>> >
>>>>>>> > I need to emit a 2D double array as key and value from
>>>>>>> mapper.There are
>>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>>> > we have to create a custom datatype.but how?I am new to these
>>>>>>> custom
>>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>>> the matrix
>>>>>>> > multiplication in a given dataset.and after that i need to emit
>>>>>>> the value
>>>>>>> > of
>>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>>> matrix.so
>>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>>> > WritableComparable.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > public class MatrixWritable implements
>>>>>>> WritableComparable<MatrixWritable>{
>>>>>>> >
>>>>>>> > /**
>>>>>>> > * @param args
>>>>>>> > */
>>>>>>> > private double[][] value;
>>>>>>> >
>>>>>>> > public MatrixWritable() {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > set(new double[0][0]);
>>>>>>> > }
>>>>>>> >
>>>>>>> > public MatrixWritable(double[][] value) {
>>>>>>> > // TODO Auto-generated constructor stub
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public void set(double[][] value) {
>>>>>>> > this.value = value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > public double[][] getValue() {
>>>>>>> > return value;
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>>> > System.out.println("write");
>>>>>>> > int row=0;
>>>>>>> > int col=0;
>>>>>>> > for(int i=0; i<value.length;i++){
>>>>>>> > row = value.length;
>>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>>> > col = value[i].length;
>>>>>>> > }
>>>>>>> > }
>>>>>>> > out.writeInt(row);
>>>>>>> > out.writeInt(col);
>>>>>>> >
>>>>>>> > System.out.println("\nTotal no of observations:
>>>>>>> "+row+":"+col);
>>>>>>> >
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> >
>>>>>>> > out.writeDouble(value[i][j]);
>>>>>>> > }
>>>>>>> > }
>>>>>>> > //priting array
>>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>>> > }
>>>>>>> > System.out.println("");
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>>> > int row = in.readInt();
>>>>>>> > int col = in.readInt();
>>>>>>> >
>>>>>>> > double[][] value = new double[row][col];
>>>>>>> > for(int i=0;i<row ; i++){
>>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>>> > value[i][j] = in.readDouble();
>>>>>>> >
>>>>>>> > }
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int hashCode() {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public boolean equals(Object o) {
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > @Override
>>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>>> > // TODO Auto-generated method stub
>>>>>>> > return 0;
>>>>>>> > }
>>>>>>> > @Override
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > return Arrays.toString(value);
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > I wrote matrix write,readfields and toString.
>>>>>>> >
>>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>>> >
>>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>>> with
>>>>>>> > emiting only custom value- for checking)
>>>>>>> >
>>>>>>> > public class MyReducer extends
>>>>>>> >
>>>>>>> >
>>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>>> >
>>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>>> > for(MatrixWritable c : values){
>>>>>>> >
>>>>>>> > System.out.println("print value "+c.toString());
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>>> toString()
>>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>>> wrong.and i
>>>>>>> > also needed to print my data asmatrix so i tried
>>>>>>> >
>>>>>>> > public String toString() {
>>>>>>> >
>>>>>>> > String separator = ", ";
>>>>>>> > StringBuffer result = new StringBuffer();
>>>>>>> >
>>>>>>> > // iterate over the first dimension
>>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>>> > // iterate over the second dimension
>>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>>> > result.append(value[i][j]);
>>>>>>> > System.out.print(value[i][j]);
>>>>>>> > result.append(separator);
>>>>>>> > }
>>>>>>> > // remove the last separator
>>>>>>> > result.setLength(result.length() - separator.length());
>>>>>>> > // add a line break.
>>>>>>> > result.append("\n");
>>>>>>> > }
>>>>>>> >
>>>>>>> >
>>>>>>> > return result.toString();
>>>>>>> >
>>>>>>> >
>>>>>>> > }
>>>>>>> >
>>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>>> datatype
>>>>>>> > CompareTo is neccessary right .
>>>>>>> >
>>>>>>> > 4.so what should i include in that methods
>>>>>>> CompareTo,hashcode,equals and
>>>>>>> > what are these methods intended for.
>>>>>>> > Any Idea.Pls suggest a solution.
>>>>>>> >
>>>>>>> > --
>>>>>>> > *Thanks & Regards*
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Unmesha Sreeveni U.B*
>>>>>>> > *
>>>>>>> > *Junior Developer
>>>>>>> > *
>>>>>>> > *Amrita Center For Cyber Security
>>>>>>> > *
>>>>>>> > *
>>>>>>> > Amritapuri.
>>>>>>> >
>>>>>>> > www.amrita.edu/cyber/
>>>>>>> > *
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Thanks & Regards*
>>>>>>
>>>>>> Unmesha Sreeveni U.B
>>>>>>
>>>>>> *Junior Developer*
>>>>>>
>>>>>> *Amrita Center For Cyber Security *
>>>>>>
>>>>>>
>>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by Mirko Kämpf <mi...@gmail.com>.
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(*Iterable<MatrixWritable>* key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
Usually a key is only one object, not an Iterable.
To make things more clear:
What is the exact k-v-pair you need in the Reducer?
One matrix is the key, and a set of (two matrices together) are used as
value in the Reducer? What I understood from your question is
KVP<Matrix,Matrix[2]>
*But you wrote also:* I need to emit a 2D double array as key and value
from mapper.
Means, you work with a k-v-pair
KVP<Matrix,Matrix>
There Matrix is 2-D matrix of double values.
I suggest:
1.) Define the MR Data Flow.
2.) Build the custom types.
3.) Test the flow (no computation)
4.) Implement logic / computation
2013/11/3 unmesha sreeveni <un...@gmail.com>
> I tried with TwoDArrayWritable too.
>
> but i tried it by emitting only one value.
>
> row = E.length;
> col = E[0].length;
> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
> // set values in myInnerArray
> for (int k1 = 0; k1 < row; k1++) {
> for(int j1=0;j1< col;j1++){
> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>
> }
> array.set(myInnerArray);
> context.write(clusterNumber, array);
>
>
>
>
>
> this is also not working for me
>
> showing NullPointerException
>
> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
> java.lang.Exception: java.lang.NullPointerException
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
> at edu.Mapper.map(Mapper.java:277)
>
>
> Mapper.java:277 : context.write(clusterNumber, array);
>
>
>
> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr Shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Can you share the code?
>>>>
>>>> sent from mobile
>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> thanks Steve Loughran and Amr Shahin
>>>>> Amr Shahin , i refered "
>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>> the same thing only. but my toString is not returning anything.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Check this out:
>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>> definitive guide" there is a really good explanation about custom data
>>>>>> types.
>>>>>>
>>>>>> Happy Halloween
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>
>>>>>> > this is my post from stackoverflow
>>>>>> > but i am not getting any response.
>>>>>> >
>>>>>> >
>>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>>> are
>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>> the matrix
>>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>>> value
>>>>>> > of
>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so
>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>> > WritableComparable.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > public class MatrixWritable implements
>>>>>> WritableComparable<MatrixWritable>{
>>>>>> >
>>>>>> > /**
>>>>>> > * @param args
>>>>>> > */
>>>>>> > private double[][] value;
>>>>>> >
>>>>>> > public MatrixWritable() {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > set(new double[0][0]);
>>>>>> > }
>>>>>> >
>>>>>> > public MatrixWritable(double[][] value) {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public void set(double[][] value) {
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public double[][] getValue() {
>>>>>> > return value;
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>> > System.out.println("write");
>>>>>> > int row=0;
>>>>>> > int col=0;
>>>>>> > for(int i=0; i<value.length;i++){
>>>>>> > row = value.length;
>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>> > col = value[i].length;
>>>>>> > }
>>>>>> > }
>>>>>> > out.writeInt(row);
>>>>>> > out.writeInt(col);
>>>>>> >
>>>>>> > System.out.println("\nTotal no of observations:
>>>>>> "+row+":"+col);
>>>>>> >
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> >
>>>>>> > out.writeDouble(value[i][j]);
>>>>>> > }
>>>>>> > }
>>>>>> > //priting array
>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>> > }
>>>>>> > System.out.println("");
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>> > int row = in.readInt();
>>>>>> > int col = in.readInt();
>>>>>> >
>>>>>> > double[][] value = new double[row][col];
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> > value[i][j] = in.readDouble();
>>>>>> >
>>>>>> > }
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public int hashCode() {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public boolean equals(Object o) {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > @Override
>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>> > // TODO Auto-generated method stub
>>>>>> > return 0;
>>>>>> > }
>>>>>> > @Override
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > return Arrays.toString(value);
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > I wrote matrix write,readfields and toString.
>>>>>> >
>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>> >
>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>> with
>>>>>> > emiting only custom value- for checking)
>>>>>> >
>>>>>> > public class MyReducer extends
>>>>>> >
>>>>>> >
>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>> >
>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>> > for(MatrixWritable c : values){
>>>>>> >
>>>>>> > System.out.println("print value "+c.toString());
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>> toString()
>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>> wrong.and i
>>>>>> > also needed to print my data asmatrix so i tried
>>>>>> >
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > String separator = ", ";
>>>>>> > StringBuffer result = new StringBuffer();
>>>>>> >
>>>>>> > // iterate over the first dimension
>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>> > // iterate over the second dimension
>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>> > result.append(value[i][j]);
>>>>>> > System.out.print(value[i][j]);
>>>>>> > result.append(separator);
>>>>>> > }
>>>>>> > // remove the last separator
>>>>>> > result.setLength(result.length() - separator.length());
>>>>>> > // add a line break.
>>>>>> > result.append("\n");
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > return result.toString();
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>> datatype
>>>>>> > CompareTo is neccessary right .
>>>>>> >
>>>>>> > 4.so what should i include in that methods
>>>>>> CompareTo,hashcode,equals and
>>>>>> > what are these methods intended for.
>>>>>> > Any Idea.Pls suggest a solution.
>>>>>> >
>>>>>> > --
>>>>>> > *Thanks & Regards*
>>>>>> > *
>>>>>> > *
>>>>>> > Unmesha Sreeveni U.B*
>>>>>> > *
>>>>>> > *Junior Developer
>>>>>> > *
>>>>>> > *Amrita Center For Cyber Security
>>>>>> > *
>>>>>> > *
>>>>>> > Amritapuri.
>>>>>> >
>>>>>> > www.amrita.edu/cyber/
>>>>>> > *
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Mirko Kämpf <mi...@gmail.com>.
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(*Iterable<MatrixWritable>* key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
Usually a key is only one object, not an Iterable.
To make things more clear:
What is the exact k-v-pair you need in the Reducer?
One matrix is the key, and a set of (two matrices together) are used as
value in the Reducer? What I understood from your question is
KVP<Matrix,Matrix[2]>
*But you wrote also:* I need to emit a 2D double array as key and value
from mapper.
Means, you work with a k-v-pair
KVP<Matrix,Matrix>
There Matrix is 2-D matrix of double values.
I suggest:
1.) Define the MR Data Flow.
2.) Build the custom types.
3.) Test the flow (no computation)
4.) Implement logic / computation
2013/11/3 unmesha sreeveni <un...@gmail.com>
> I tried with TwoDArrayWritable too.
>
> but i tried it by emitting only one value.
>
> row = E.length;
> col = E[0].length;
> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
> // set values in myInnerArray
> for (int k1 = 0; k1 < row; k1++) {
> for(int j1=0;j1< col;j1++){
> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>
> }
> array.set(myInnerArray);
> context.write(clusterNumber, array);
>
>
>
>
>
> this is also not working for me
>
> showing NullPointerException
>
> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
> java.lang.Exception: java.lang.NullPointerException
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
> at edu.Mapper.map(Mapper.java:277)
>
>
> Mapper.java:277 : context.write(clusterNumber, array);
>
>
>
> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr Shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Can you share the code?
>>>>
>>>> sent from mobile
>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> thanks Steve Loughran and Amr Shahin
>>>>> Amr Shahin , i refered "
>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>> the same thing only. but my toString is not returning anything.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Check this out:
>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>> definitive guide" there is a really good explanation about custom data
>>>>>> types.
>>>>>>
>>>>>> Happy Halloween
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>
>>>>>> > this is my post from stackoverflow
>>>>>> > but i am not getting any response.
>>>>>> >
>>>>>> >
>>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>>> are
>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>> the matrix
>>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>>> value
>>>>>> > of
>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so
>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>> > WritableComparable.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > public class MatrixWritable implements
>>>>>> WritableComparable<MatrixWritable>{
>>>>>> >
>>>>>> > /**
>>>>>> > * @param args
>>>>>> > */
>>>>>> > private double[][] value;
>>>>>> >
>>>>>> > public MatrixWritable() {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > set(new double[0][0]);
>>>>>> > }
>>>>>> >
>>>>>> > public MatrixWritable(double[][] value) {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public void set(double[][] value) {
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public double[][] getValue() {
>>>>>> > return value;
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>> > System.out.println("write");
>>>>>> > int row=0;
>>>>>> > int col=0;
>>>>>> > for(int i=0; i<value.length;i++){
>>>>>> > row = value.length;
>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>> > col = value[i].length;
>>>>>> > }
>>>>>> > }
>>>>>> > out.writeInt(row);
>>>>>> > out.writeInt(col);
>>>>>> >
>>>>>> > System.out.println("\nTotal no of observations:
>>>>>> "+row+":"+col);
>>>>>> >
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> >
>>>>>> > out.writeDouble(value[i][j]);
>>>>>> > }
>>>>>> > }
>>>>>> > //priting array
>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>> > }
>>>>>> > System.out.println("");
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>> > int row = in.readInt();
>>>>>> > int col = in.readInt();
>>>>>> >
>>>>>> > double[][] value = new double[row][col];
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> > value[i][j] = in.readDouble();
>>>>>> >
>>>>>> > }
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public int hashCode() {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public boolean equals(Object o) {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > @Override
>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>> > // TODO Auto-generated method stub
>>>>>> > return 0;
>>>>>> > }
>>>>>> > @Override
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > return Arrays.toString(value);
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > I wrote matrix write,readfields and toString.
>>>>>> >
>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>> >
>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>> with
>>>>>> > emiting only custom value- for checking)
>>>>>> >
>>>>>> > public class MyReducer extends
>>>>>> >
>>>>>> >
>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>> >
>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>> > for(MatrixWritable c : values){
>>>>>> >
>>>>>> > System.out.println("print value "+c.toString());
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>> toString()
>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>> wrong.and i
>>>>>> > also needed to print my data asmatrix so i tried
>>>>>> >
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > String separator = ", ";
>>>>>> > StringBuffer result = new StringBuffer();
>>>>>> >
>>>>>> > // iterate over the first dimension
>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>> > // iterate over the second dimension
>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>> > result.append(value[i][j]);
>>>>>> > System.out.print(value[i][j]);
>>>>>> > result.append(separator);
>>>>>> > }
>>>>>> > // remove the last separator
>>>>>> > result.setLength(result.length() - separator.length());
>>>>>> > // add a line break.
>>>>>> > result.append("\n");
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > return result.toString();
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>> datatype
>>>>>> > CompareTo is neccessary right .
>>>>>> >
>>>>>> > 4.so what should i include in that methods
>>>>>> CompareTo,hashcode,equals and
>>>>>> > what are these methods intended for.
>>>>>> > Any Idea.Pls suggest a solution.
>>>>>> >
>>>>>> > --
>>>>>> > *Thanks & Regards*
>>>>>> > *
>>>>>> > *
>>>>>> > Unmesha Sreeveni U.B*
>>>>>> > *
>>>>>> > *Junior Developer
>>>>>> > *
>>>>>> > *Amrita Center For Cyber Security
>>>>>> > *
>>>>>> > *
>>>>>> > Amritapuri.
>>>>>> >
>>>>>> > www.amrita.edu/cyber/
>>>>>> > *
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Mirko Kämpf <mi...@gmail.com>.
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(*Iterable<MatrixWritable>* key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
Usually a key is only one object, not an Iterable.
To make things more clear:
What is the exact k-v-pair you need in the Reducer?
One matrix is the key, and a set of (two matrices together) are used as
value in the Reducer? What I understood from your question is
KVP<Matrix,Matrix[2]>
*But you wrote also:* I need to emit a 2D double array as key and value
from mapper.
Means, you work with a k-v-pair
KVP<Matrix,Matrix>
There Matrix is 2-D matrix of double values.
I suggest:
1.) Define the MR Data Flow.
2.) Build the custom types.
3.) Test the flow (no computation)
4.) Implement logic / computation
2013/11/3 unmesha sreeveni <un...@gmail.com>
> I tried with TwoDArrayWritable too.
>
> but i tried it by emitting only one value.
>
> row = E.length;
> col = E[0].length;
> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
> // set values in myInnerArray
> for (int k1 = 0; k1 < row; k1++) {
> for(int j1=0;j1< col;j1++){
> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>
> }
> array.set(myInnerArray);
> context.write(clusterNumber, array);
>
>
>
>
>
> this is also not working for me
>
> showing NullPointerException
>
> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
> java.lang.Exception: java.lang.NullPointerException
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
> at edu.Mapper.map(Mapper.java:277)
>
>
> Mapper.java:277 : context.write(clusterNumber, array);
>
>
>
> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr Shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Can you share the code?
>>>>
>>>> sent from mobile
>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> thanks Steve Loughran and Amr Shahin
>>>>> Amr Shahin , i refered "
>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>> the same thing only. but my toString is not returning anything.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Check this out:
>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>> definitive guide" there is a really good explanation about custom data
>>>>>> types.
>>>>>>
>>>>>> Happy Halloween
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>
>>>>>> > this is my post from stackoverflow
>>>>>> > but i am not getting any response.
>>>>>> >
>>>>>> >
>>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>>> are
>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>> the matrix
>>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>>> value
>>>>>> > of
>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so
>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>> > WritableComparable.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > public class MatrixWritable implements
>>>>>> WritableComparable<MatrixWritable>{
>>>>>> >
>>>>>> > /**
>>>>>> > * @param args
>>>>>> > */
>>>>>> > private double[][] value;
>>>>>> >
>>>>>> > public MatrixWritable() {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > set(new double[0][0]);
>>>>>> > }
>>>>>> >
>>>>>> > public MatrixWritable(double[][] value) {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public void set(double[][] value) {
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public double[][] getValue() {
>>>>>> > return value;
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>> > System.out.println("write");
>>>>>> > int row=0;
>>>>>> > int col=0;
>>>>>> > for(int i=0; i<value.length;i++){
>>>>>> > row = value.length;
>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>> > col = value[i].length;
>>>>>> > }
>>>>>> > }
>>>>>> > out.writeInt(row);
>>>>>> > out.writeInt(col);
>>>>>> >
>>>>>> > System.out.println("\nTotal no of observations:
>>>>>> "+row+":"+col);
>>>>>> >
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> >
>>>>>> > out.writeDouble(value[i][j]);
>>>>>> > }
>>>>>> > }
>>>>>> > //priting array
>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>> > }
>>>>>> > System.out.println("");
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>> > int row = in.readInt();
>>>>>> > int col = in.readInt();
>>>>>> >
>>>>>> > double[][] value = new double[row][col];
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> > value[i][j] = in.readDouble();
>>>>>> >
>>>>>> > }
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public int hashCode() {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public boolean equals(Object o) {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > @Override
>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>> > // TODO Auto-generated method stub
>>>>>> > return 0;
>>>>>> > }
>>>>>> > @Override
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > return Arrays.toString(value);
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > I wrote matrix write,readfields and toString.
>>>>>> >
>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>> >
>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>> with
>>>>>> > emiting only custom value- for checking)
>>>>>> >
>>>>>> > public class MyReducer extends
>>>>>> >
>>>>>> >
>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>> >
>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>> > for(MatrixWritable c : values){
>>>>>> >
>>>>>> > System.out.println("print value "+c.toString());
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>> toString()
>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>> wrong.and i
>>>>>> > also needed to print my data asmatrix so i tried
>>>>>> >
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > String separator = ", ";
>>>>>> > StringBuffer result = new StringBuffer();
>>>>>> >
>>>>>> > // iterate over the first dimension
>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>> > // iterate over the second dimension
>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>> > result.append(value[i][j]);
>>>>>> > System.out.print(value[i][j]);
>>>>>> > result.append(separator);
>>>>>> > }
>>>>>> > // remove the last separator
>>>>>> > result.setLength(result.length() - separator.length());
>>>>>> > // add a line break.
>>>>>> > result.append("\n");
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > return result.toString();
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>> datatype
>>>>>> > CompareTo is neccessary right .
>>>>>> >
>>>>>> > 4.so what should i include in that methods
>>>>>> CompareTo,hashcode,equals and
>>>>>> > what are these methods intended for.
>>>>>> > Any Idea.Pls suggest a solution.
>>>>>> >
>>>>>> > --
>>>>>> > *Thanks & Regards*
>>>>>> > *
>>>>>> > *
>>>>>> > Unmesha Sreeveni U.B*
>>>>>> > *
>>>>>> > *Junior Developer
>>>>>> > *
>>>>>> > *Amrita Center For Cyber Security
>>>>>> > *
>>>>>> > *
>>>>>> > Amritapuri.
>>>>>> >
>>>>>> > www.amrita.edu/cyber/
>>>>>> > *
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Mirko Kämpf <mi...@gmail.com>.
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(*Iterable<MatrixWritable>* key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
Usually a key is only one object, not an Iterable.
To make things more clear:
What is the exact k-v-pair you need in the Reducer?
One matrix is the key, and a set of (two matrices together) are used as
value in the Reducer? What I understood from your question is
KVP<Matrix,Matrix[2]>
*But you wrote also:* I need to emit a 2D double array as key and value
from mapper.
Means, you work with a k-v-pair
KVP<Matrix,Matrix>
There Matrix is 2-D matrix of double values.
I suggest:
1.) Define the MR Data Flow.
2.) Build the custom types.
3.) Test the flow (no computation)
4.) Implement logic / computation
2013/11/3 unmesha sreeveni <un...@gmail.com>
> I tried with TwoDArrayWritable too.
>
> but i tried it by emitting only one value.
>
> row = E.length;
> col = E[0].length;
> TwoDArrayWritable array = new TwoDArrayWritable (DoubleWritable.class);
> DoubleWritable[][] myInnerArray = new DoubleWritable[row][col];
> // set values in myInnerArray
> for (int k1 = 0; k1 < row; k1++) {
> for(int j1=0;j1< col;j1++){
> myInnerArray[k1][j1] = new DoubleWritable(E[k1][j1]);
>
> }
> array.set(myInnerArray);
> context.write(clusterNumber, array);
>
>
>
>
>
> this is also not working for me
>
> showing NullPointerException
>
> 13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
> java.lang.Exception: java.lang.NullPointerException
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
> at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
> at edu.Mapper.map(Mapper.java:277)
>
>
> Mapper.java:277 : context.write(clusterNumber, array);
>
>
>
> On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr Shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>>
>>> @Amr shahin
>>> this is my post from stackoverflow
>>> but i am not getting any response.
>>>
>>>
>>> I need to emit a 2D double array as key and value from mapper.There are
>>> questions posted in stackoverflow. But they are not answered.
>>> we have to create a custom datatype.but how?I am new to these custom
>>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>>> multiplication in a given dataset.and after that i need to emit the value of
>>> A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so how to emit these matrices from mapper.And the value should
>>> be corresponding to the key itself.I think for that we need to use
>>> WritableComparable.
>>>
>>>
>>>
>>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>>
>>> /**
>>> * @param args
>>> */
>>> private double[][] value;
>>>
>>> public MatrixWritable() {
>>> // TODO Auto-generated constructor stub
>>> set(new double[0][0]);
>>> }
>>>
>>> public MatrixWritable(double[][] value) {
>>> // TODO Auto-generated constructor stub
>>> this.value = value;
>>> }
>>>
>>> public void set(double[][] value) {
>>> this.value = value;
>>> }
>>>
>>> public double[][] getValue() {
>>> return value;
>>> }
>>>
>>> @Override
>>> public void write(DataOutput out) throws IOException {
>>> System.out.println("write");
>>> int row=0;
>>> int col=0;
>>> for(int i=0; i<value.length;i++){
>>> row = value.length;
>>> for(int j=0; j<value[i].length; j++){
>>> col = value[i].length;
>>> }
>>> }
>>> out.writeInt(row);
>>> out.writeInt(col);
>>>
>>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>>
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>>
>>> out.writeDouble(value[i][j]);
>>> }
>>> }
>>> //priting array
>>> for(int vali =0;vali< value.length ;vali ++){
>>> for(int valj = 0;valj <value[0].length;valj++){
>>> System.out.print(value[vali][valj]+ "\t");
>>> }
>>> System.out.println("");
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public void readFields(DataInput in) throws IOException {
>>> int row = in.readInt();
>>> int col = in.readInt();
>>>
>>> double[][] value = new double[row][col];
>>> for(int i=0;i<row ; i++){
>>> for(int j= 0 ; j< col;j++){
>>> value[i][j] = in.readDouble();
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> @Override
>>> public int hashCode() {
>>>
>>> }
>>>
>>> @Override
>>> public boolean equals(Object o) {
>>>
>>> }
>>>
>>>
>>> @Override
>>> public int compareTo(MatrixWritable o) {
>>> // TODO Auto-generated method stub
>>> return 0;
>>> }
>>> @Override
>>> public String toString() {
>>>
>>> return Arrays.toString(value);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>> I wrote matrix write,readfields and toString.
>>>
>>> 1.But my toString is not returning anything. why is it so?
>>>
>>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>>
>>> public class MyReducer extends
>>>
>>>
>>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>
>>> public void reduce(Iterable<MatrixWritable> key,
>>> Iterable<MatrixWritable> values, Context context){
>>> for(MatrixWritable c : values){
>>>
>>> System.out.println("print value "+c.toString());
>>>
>>> }
>>>
>>> }
>>>
>>> but Nothing is printing.when i tried to print value[0].length in toString()
>>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>>> i also needed to print my data asmatrix so i tried
>>>
>>> public String toString() {
>>>
>>> String separator = ", ";
>>> StringBuffer result = new StringBuffer();
>>>
>>> // iterate over the first dimension
>>> for (int i = 0; i < value.length; i++) {
>>> // iterate over the second dimension
>>> for(int j = 0; j < value[i].length; j++){
>>> result.append(value[i][j]);
>>> System.out.print(value[i][j]);
>>> result.append(separator);
>>> }
>>> // remove the last separator
>>> result.setLength(result.length() - separator.length());
>>> // add a line break.
>>> result.append("\n");
>>> }
>>>
>>>
>>> return result.toString();
>>>
>>>
>>> }
>>>
>>>
>>>
>>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Can you share the code?
>>>>
>>>> sent from mobile
>>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> thanks Steve Loughran and Amr Shahin
>>>>> Amr Shahin , i refered "
>>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>>> the same thing only. but my toString is not returning anything.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>>
>>>>>> Check this out:
>>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>>> definitive guide" there is a really good explanation about custom data
>>>>>> types.
>>>>>>
>>>>>> Happy Halloween
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>>> unmeshabiju@gmail.com>wrote:
>>>>>>
>>>>>> > this is my post from stackoverflow
>>>>>> > but i am not getting any response.
>>>>>> >
>>>>>> >
>>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>>> are
>>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>>> > datatypes. i dnt have any idea where to start.I am doing some of
>>>>>> the matrix
>>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>>> value
>>>>>> > of
>>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>>> matrix.so
>>>>>> > how to emit these matrices from mapper.And the value should be
>>>>>> > corresponding to the key itself.I think for that we need to use
>>>>>> > WritableComparable.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > public class MatrixWritable implements
>>>>>> WritableComparable<MatrixWritable>{
>>>>>> >
>>>>>> > /**
>>>>>> > * @param args
>>>>>> > */
>>>>>> > private double[][] value;
>>>>>> >
>>>>>> > public MatrixWritable() {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > set(new double[0][0]);
>>>>>> > }
>>>>>> >
>>>>>> > public MatrixWritable(double[][] value) {
>>>>>> > // TODO Auto-generated constructor stub
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public void set(double[][] value) {
>>>>>> > this.value = value;
>>>>>> > }
>>>>>> >
>>>>>> > public double[][] getValue() {
>>>>>> > return value;
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void write(DataOutput out) throws IOException {
>>>>>> > System.out.println("write");
>>>>>> > int row=0;
>>>>>> > int col=0;
>>>>>> > for(int i=0; i<value.length;i++){
>>>>>> > row = value.length;
>>>>>> > for(int j=0; j<value[i].length; j++){
>>>>>> > col = value[i].length;
>>>>>> > }
>>>>>> > }
>>>>>> > out.writeInt(row);
>>>>>> > out.writeInt(col);
>>>>>> >
>>>>>> > System.out.println("\nTotal no of observations:
>>>>>> "+row+":"+col);
>>>>>> >
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> >
>>>>>> > out.writeDouble(value[i][j]);
>>>>>> > }
>>>>>> > }
>>>>>> > //priting array
>>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>>> > }
>>>>>> > System.out.println("");
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public void readFields(DataInput in) throws IOException {
>>>>>> > int row = in.readInt();
>>>>>> > int col = in.readInt();
>>>>>> >
>>>>>> > double[][] value = new double[row][col];
>>>>>> > for(int i=0;i<row ; i++){
>>>>>> > for(int j= 0 ; j< col;j++){
>>>>>> > value[i][j] = in.readDouble();
>>>>>> >
>>>>>> > }
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public int hashCode() {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > @Override
>>>>>> > public boolean equals(Object o) {
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > @Override
>>>>>> > public int compareTo(MatrixWritable o) {
>>>>>> > // TODO Auto-generated method stub
>>>>>> > return 0;
>>>>>> > }
>>>>>> > @Override
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > return Arrays.toString(value);
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > I wrote matrix write,readfields and toString.
>>>>>> >
>>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>>> >
>>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried
>>>>>> with
>>>>>> > emiting only custom value- for checking)
>>>>>> >
>>>>>> > public class MyReducer extends
>>>>>> >
>>>>>> >
>>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>>> >
>>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>>> > for(MatrixWritable c : values){
>>>>>> >
>>>>>> > System.out.println("print value "+c.toString());
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>>> toString()
>>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>>> wrong.and i
>>>>>> > also needed to print my data asmatrix so i tried
>>>>>> >
>>>>>> > public String toString() {
>>>>>> >
>>>>>> > String separator = ", ";
>>>>>> > StringBuffer result = new StringBuffer();
>>>>>> >
>>>>>> > // iterate over the first dimension
>>>>>> > for (int i = 0; i < value.length; i++) {
>>>>>> > // iterate over the second dimension
>>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>>> > result.append(value[i][j]);
>>>>>> > System.out.print(value[i][j]);
>>>>>> > result.append(separator);
>>>>>> > }
>>>>>> > // remove the last separator
>>>>>> > result.setLength(result.length() - separator.length());
>>>>>> > // add a line break.
>>>>>> > result.append("\n");
>>>>>> > }
>>>>>> >
>>>>>> >
>>>>>> > return result.toString();
>>>>>> >
>>>>>> >
>>>>>> > }
>>>>>> >
>>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>>> datatype
>>>>>> > CompareTo is neccessary right .
>>>>>> >
>>>>>> > 4.so what should i include in that methods
>>>>>> CompareTo,hashcode,equals and
>>>>>> > what are these methods intended for.
>>>>>> > Any Idea.Pls suggest a solution.
>>>>>> >
>>>>>> > --
>>>>>> > *Thanks & Regards*
>>>>>> > *
>>>>>> > *
>>>>>> > Unmesha Sreeveni U.B*
>>>>>> > *
>>>>>> > *Junior Developer
>>>>>> > *
>>>>>> > *Amrita Center For Cyber Security
>>>>>> > *
>>>>>> > *
>>>>>> > Amritapuri.
>>>>>> >
>>>>>> > www.amrita.edu/cyber/
>>>>>> > *
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>> *Amrita Center For Cyber Security *
>>>>>
>>>>>
>>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
I tried with TwoDArrayWritable too.
but i tried it by emitting only one value.
row = E.length;
col = E[0].length;
TwoDArrayWritable array = new TwoDArrayWritable
(DoubleWritable.class);
DoubleWritable[][] myInnerArray = new
DoubleWritable[row][col];
// set values in myInnerArray
for (int k1 = 0; k1 < row; k1++) {
for(int j1=0;j1< col;j1++){
myInnerArray[k1][j1] = new
DoubleWritable(E[k1][j1]);
}
array.set(myInnerArray);
context.write(clusterNumber, array);
this is also not working for me
showing NullPointerException
13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
at edu.Mapper.map(Mapper.java:277)
Mapper.java:277 : context.write(clusterNumber, array);
On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr Shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Can you share the code?
>>>
>>> sent from mobile
>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> thanks Steve Loughran and Amr Shahin
>>>> Amr Shahin , i refered "
>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>> the same thing only. but my toString is not returning anything.
>>>>
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>
>>>>> Check this out:
>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>> definitive guide" there is a really good explanation about custom data
>>>>> types.
>>>>>
>>>>> Happy Halloween
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com>wrote:
>>>>>
>>>>> > this is my post from stackoverflow
>>>>> > but i am not getting any response.
>>>>> >
>>>>> >
>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>> are
>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>>> matrix
>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>> value
>>>>> > of
>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so
>>>>> > how to emit these matrices from mapper.And the value should be
>>>>> > corresponding to the key itself.I think for that we need to use
>>>>> > WritableComparable.
>>>>> >
>>>>> >
>>>>> >
>>>>> > public class MatrixWritable implements
>>>>> WritableComparable<MatrixWritable>{
>>>>> >
>>>>> > /**
>>>>> > * @param args
>>>>> > */
>>>>> > private double[][] value;
>>>>> >
>>>>> > public MatrixWritable() {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > set(new double[0][0]);
>>>>> > }
>>>>> >
>>>>> > public MatrixWritable(double[][] value) {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public void set(double[][] value) {
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public double[][] getValue() {
>>>>> > return value;
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void write(DataOutput out) throws IOException {
>>>>> > System.out.println("write");
>>>>> > int row=0;
>>>>> > int col=0;
>>>>> > for(int i=0; i<value.length;i++){
>>>>> > row = value.length;
>>>>> > for(int j=0; j<value[i].length; j++){
>>>>> > col = value[i].length;
>>>>> > }
>>>>> > }
>>>>> > out.writeInt(row);
>>>>> > out.writeInt(col);
>>>>> >
>>>>> > System.out.println("\nTotal no of observations:
>>>>> "+row+":"+col);
>>>>> >
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> >
>>>>> > out.writeDouble(value[i][j]);
>>>>> > }
>>>>> > }
>>>>> > //priting array
>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>> > }
>>>>> > System.out.println("");
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void readFields(DataInput in) throws IOException {
>>>>> > int row = in.readInt();
>>>>> > int col = in.readInt();
>>>>> >
>>>>> > double[][] value = new double[row][col];
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> > value[i][j] = in.readDouble();
>>>>> >
>>>>> > }
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public int hashCode() {
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public boolean equals(Object o) {
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> > @Override
>>>>> > public int compareTo(MatrixWritable o) {
>>>>> > // TODO Auto-generated method stub
>>>>> > return 0;
>>>>> > }
>>>>> > @Override
>>>>> > public String toString() {
>>>>> >
>>>>> > return Arrays.toString(value);
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > I wrote matrix write,readfields and toString.
>>>>> >
>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>> >
>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>>> > emiting only custom value- for checking)
>>>>> >
>>>>> > public class MyReducer extends
>>>>> >
>>>>> >
>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>> >
>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>> > for(MatrixWritable c : values){
>>>>> >
>>>>> > System.out.println("print value "+c.toString());
>>>>> >
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>> toString()
>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>> wrong.and i
>>>>> > also needed to print my data asmatrix so i tried
>>>>> >
>>>>> > public String toString() {
>>>>> >
>>>>> > String separator = ", ";
>>>>> > StringBuffer result = new StringBuffer();
>>>>> >
>>>>> > // iterate over the first dimension
>>>>> > for (int i = 0; i < value.length; i++) {
>>>>> > // iterate over the second dimension
>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>> > result.append(value[i][j]);
>>>>> > System.out.print(value[i][j]);
>>>>> > result.append(separator);
>>>>> > }
>>>>> > // remove the last separator
>>>>> > result.setLength(result.length() - separator.length());
>>>>> > // add a line break.
>>>>> > result.append("\n");
>>>>> > }
>>>>> >
>>>>> >
>>>>> > return result.toString();
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>> datatype
>>>>> > CompareTo is neccessary right .
>>>>> >
>>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>>> and
>>>>> > what are these methods intended for.
>>>>> > Any Idea.Pls suggest a solution.
>>>>> >
>>>>> > --
>>>>> > *Thanks & Regards*
>>>>> > *
>>>>> > *
>>>>> > Unmesha Sreeveni U.B*
>>>>> > *
>>>>> > *Junior Developer
>>>>> > *
>>>>> > *Amrita Center For Cyber Security
>>>>> > *
>>>>> > *
>>>>> > Amritapuri.
>>>>> >
>>>>> > www.amrita.edu/cyber/
>>>>> > *
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
I tried with TwoDArrayWritable too.
but i tried it by emitting only one value.
row = E.length;
col = E[0].length;
TwoDArrayWritable array = new TwoDArrayWritable
(DoubleWritable.class);
DoubleWritable[][] myInnerArray = new
DoubleWritable[row][col];
// set values in myInnerArray
for (int k1 = 0; k1 < row; k1++) {
for(int j1=0;j1< col;j1++){
myInnerArray[k1][j1] = new
DoubleWritable(E[k1][j1]);
}
array.set(myInnerArray);
context.write(clusterNumber, array);
this is also not working for me
showing NullPointerException
13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
at edu.Mapper.map(Mapper.java:277)
Mapper.java:277 : context.write(clusterNumber, array);
On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr Shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Can you share the code?
>>>
>>> sent from mobile
>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> thanks Steve Loughran and Amr Shahin
>>>> Amr Shahin , i refered "
>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>> the same thing only. but my toString is not returning anything.
>>>>
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>
>>>>> Check this out:
>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>> definitive guide" there is a really good explanation about custom data
>>>>> types.
>>>>>
>>>>> Happy Halloween
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com>wrote:
>>>>>
>>>>> > this is my post from stackoverflow
>>>>> > but i am not getting any response.
>>>>> >
>>>>> >
>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>> are
>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>>> matrix
>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>> value
>>>>> > of
>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so
>>>>> > how to emit these matrices from mapper.And the value should be
>>>>> > corresponding to the key itself.I think for that we need to use
>>>>> > WritableComparable.
>>>>> >
>>>>> >
>>>>> >
>>>>> > public class MatrixWritable implements
>>>>> WritableComparable<MatrixWritable>{
>>>>> >
>>>>> > /**
>>>>> > * @param args
>>>>> > */
>>>>> > private double[][] value;
>>>>> >
>>>>> > public MatrixWritable() {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > set(new double[0][0]);
>>>>> > }
>>>>> >
>>>>> > public MatrixWritable(double[][] value) {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public void set(double[][] value) {
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public double[][] getValue() {
>>>>> > return value;
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void write(DataOutput out) throws IOException {
>>>>> > System.out.println("write");
>>>>> > int row=0;
>>>>> > int col=0;
>>>>> > for(int i=0; i<value.length;i++){
>>>>> > row = value.length;
>>>>> > for(int j=0; j<value[i].length; j++){
>>>>> > col = value[i].length;
>>>>> > }
>>>>> > }
>>>>> > out.writeInt(row);
>>>>> > out.writeInt(col);
>>>>> >
>>>>> > System.out.println("\nTotal no of observations:
>>>>> "+row+":"+col);
>>>>> >
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> >
>>>>> > out.writeDouble(value[i][j]);
>>>>> > }
>>>>> > }
>>>>> > //priting array
>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>> > }
>>>>> > System.out.println("");
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void readFields(DataInput in) throws IOException {
>>>>> > int row = in.readInt();
>>>>> > int col = in.readInt();
>>>>> >
>>>>> > double[][] value = new double[row][col];
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> > value[i][j] = in.readDouble();
>>>>> >
>>>>> > }
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public int hashCode() {
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public boolean equals(Object o) {
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> > @Override
>>>>> > public int compareTo(MatrixWritable o) {
>>>>> > // TODO Auto-generated method stub
>>>>> > return 0;
>>>>> > }
>>>>> > @Override
>>>>> > public String toString() {
>>>>> >
>>>>> > return Arrays.toString(value);
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > I wrote matrix write,readfields and toString.
>>>>> >
>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>> >
>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>>> > emiting only custom value- for checking)
>>>>> >
>>>>> > public class MyReducer extends
>>>>> >
>>>>> >
>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>> >
>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>> > for(MatrixWritable c : values){
>>>>> >
>>>>> > System.out.println("print value "+c.toString());
>>>>> >
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>> toString()
>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>> wrong.and i
>>>>> > also needed to print my data asmatrix so i tried
>>>>> >
>>>>> > public String toString() {
>>>>> >
>>>>> > String separator = ", ";
>>>>> > StringBuffer result = new StringBuffer();
>>>>> >
>>>>> > // iterate over the first dimension
>>>>> > for (int i = 0; i < value.length; i++) {
>>>>> > // iterate over the second dimension
>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>> > result.append(value[i][j]);
>>>>> > System.out.print(value[i][j]);
>>>>> > result.append(separator);
>>>>> > }
>>>>> > // remove the last separator
>>>>> > result.setLength(result.length() - separator.length());
>>>>> > // add a line break.
>>>>> > result.append("\n");
>>>>> > }
>>>>> >
>>>>> >
>>>>> > return result.toString();
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>> datatype
>>>>> > CompareTo is neccessary right .
>>>>> >
>>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>>> and
>>>>> > what are these methods intended for.
>>>>> > Any Idea.Pls suggest a solution.
>>>>> >
>>>>> > --
>>>>> > *Thanks & Regards*
>>>>> > *
>>>>> > *
>>>>> > Unmesha Sreeveni U.B*
>>>>> > *
>>>>> > *Junior Developer
>>>>> > *
>>>>> > *Amrita Center For Cyber Security
>>>>> > *
>>>>> > *
>>>>> > Amritapuri.
>>>>> >
>>>>> > www.amrita.edu/cyber/
>>>>> > *
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
I tried with TwoDArrayWritable too.
but i tried it by emitting only one value.
row = E.length;
col = E[0].length;
TwoDArrayWritable array = new TwoDArrayWritable
(DoubleWritable.class);
DoubleWritable[][] myInnerArray = new
DoubleWritable[row][col];
// set values in myInnerArray
for (int k1 = 0; k1 < row; k1++) {
for(int j1=0;j1< col;j1++){
myInnerArray[k1][j1] = new
DoubleWritable(E[k1][j1]);
}
array.set(myInnerArray);
context.write(clusterNumber, array);
this is also not working for me
showing NullPointerException
13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
at edu.Mapper.map(Mapper.java:277)
Mapper.java:277 : context.write(clusterNumber, array);
On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr Shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Can you share the code?
>>>
>>> sent from mobile
>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> thanks Steve Loughran and Amr Shahin
>>>> Amr Shahin , i refered "
>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>> the same thing only. but my toString is not returning anything.
>>>>
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>
>>>>> Check this out:
>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>> definitive guide" there is a really good explanation about custom data
>>>>> types.
>>>>>
>>>>> Happy Halloween
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com>wrote:
>>>>>
>>>>> > this is my post from stackoverflow
>>>>> > but i am not getting any response.
>>>>> >
>>>>> >
>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>> are
>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>>> matrix
>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>> value
>>>>> > of
>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so
>>>>> > how to emit these matrices from mapper.And the value should be
>>>>> > corresponding to the key itself.I think for that we need to use
>>>>> > WritableComparable.
>>>>> >
>>>>> >
>>>>> >
>>>>> > public class MatrixWritable implements
>>>>> WritableComparable<MatrixWritable>{
>>>>> >
>>>>> > /**
>>>>> > * @param args
>>>>> > */
>>>>> > private double[][] value;
>>>>> >
>>>>> > public MatrixWritable() {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > set(new double[0][0]);
>>>>> > }
>>>>> >
>>>>> > public MatrixWritable(double[][] value) {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public void set(double[][] value) {
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public double[][] getValue() {
>>>>> > return value;
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void write(DataOutput out) throws IOException {
>>>>> > System.out.println("write");
>>>>> > int row=0;
>>>>> > int col=0;
>>>>> > for(int i=0; i<value.length;i++){
>>>>> > row = value.length;
>>>>> > for(int j=0; j<value[i].length; j++){
>>>>> > col = value[i].length;
>>>>> > }
>>>>> > }
>>>>> > out.writeInt(row);
>>>>> > out.writeInt(col);
>>>>> >
>>>>> > System.out.println("\nTotal no of observations:
>>>>> "+row+":"+col);
>>>>> >
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> >
>>>>> > out.writeDouble(value[i][j]);
>>>>> > }
>>>>> > }
>>>>> > //priting array
>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>> > }
>>>>> > System.out.println("");
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void readFields(DataInput in) throws IOException {
>>>>> > int row = in.readInt();
>>>>> > int col = in.readInt();
>>>>> >
>>>>> > double[][] value = new double[row][col];
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> > value[i][j] = in.readDouble();
>>>>> >
>>>>> > }
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public int hashCode() {
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public boolean equals(Object o) {
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> > @Override
>>>>> > public int compareTo(MatrixWritable o) {
>>>>> > // TODO Auto-generated method stub
>>>>> > return 0;
>>>>> > }
>>>>> > @Override
>>>>> > public String toString() {
>>>>> >
>>>>> > return Arrays.toString(value);
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > I wrote matrix write,readfields and toString.
>>>>> >
>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>> >
>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>>> > emiting only custom value- for checking)
>>>>> >
>>>>> > public class MyReducer extends
>>>>> >
>>>>> >
>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>> >
>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>> > for(MatrixWritable c : values){
>>>>> >
>>>>> > System.out.println("print value "+c.toString());
>>>>> >
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>> toString()
>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>> wrong.and i
>>>>> > also needed to print my data asmatrix so i tried
>>>>> >
>>>>> > public String toString() {
>>>>> >
>>>>> > String separator = ", ";
>>>>> > StringBuffer result = new StringBuffer();
>>>>> >
>>>>> > // iterate over the first dimension
>>>>> > for (int i = 0; i < value.length; i++) {
>>>>> > // iterate over the second dimension
>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>> > result.append(value[i][j]);
>>>>> > System.out.print(value[i][j]);
>>>>> > result.append(separator);
>>>>> > }
>>>>> > // remove the last separator
>>>>> > result.setLength(result.length() - separator.length());
>>>>> > // add a line break.
>>>>> > result.append("\n");
>>>>> > }
>>>>> >
>>>>> >
>>>>> > return result.toString();
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>> datatype
>>>>> > CompareTo is neccessary right .
>>>>> >
>>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>>> and
>>>>> > what are these methods intended for.
>>>>> > Any Idea.Pls suggest a solution.
>>>>> >
>>>>> > --
>>>>> > *Thanks & Regards*
>>>>> > *
>>>>> > *
>>>>> > Unmesha Sreeveni U.B*
>>>>> > *
>>>>> > *Junior Developer
>>>>> > *
>>>>> > *Amrita Center For Cyber Security
>>>>> > *
>>>>> > *
>>>>> > Amritapuri.
>>>>> >
>>>>> > www.amrita.edu/cyber/
>>>>> > *
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
I tried with TwoDArrayWritable too.
but i tried it by emitting only one value.
row = E.length;
col = E[0].length;
TwoDArrayWritable array = new TwoDArrayWritable
(DoubleWritable.class);
DoubleWritable[][] myInnerArray = new
DoubleWritable[row][col];
// set values in myInnerArray
for (int k1 = 0; k1 < row; k1++) {
for(int j1=0;j1< col;j1++){
myInnerArray[k1][j1] = new
DoubleWritable(E[k1][j1]);
}
array.set(myInnerArray);
context.write(clusterNumber, array);
this is also not working for me
showing NullPointerException
13/11/01 16:34:07 INFO mapred.LocalJobRunner: Map task executor complete.
13/11/01 16:34:07 WARN mapred.LocalJobRunner: job_local724758890_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.TwoDArrayWritable.write(TwoDArrayWritable.java:91)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:100)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:84)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:945)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:601)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
at edu.Mapper.map(Mapper.java:277)
Mapper.java:277 : context.write(clusterNumber, array);
On Sun, Nov 3, 2013 at 5:07 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr Shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
>
>> @Amr shahin
>> this is my post from stackoverflow
>> but i am not getting any response.
>>
>>
>> I need to emit a 2D double array as key and value from mapper.There are
>> questions posted in stackoverflow. But they are not answered.
>> we have to create a custom datatype.but how?I am new to these custom
>> datatypes. i dnt have any idea where to start.I am doing some of the matrix
>> multiplication in a given dataset.and after that i need to emit the value of
>> A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so how to emit these matrices from mapper.And the value should be
>> corresponding to the key itself.I think for that we need to use
>> WritableComparable.
>>
>>
>>
>> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>>
>> /**
>> * @param args
>> */
>> private double[][] value;
>>
>> public MatrixWritable() {
>> // TODO Auto-generated constructor stub
>> set(new double[0][0]);
>> }
>>
>> public MatrixWritable(double[][] value) {
>> // TODO Auto-generated constructor stub
>> this.value = value;
>> }
>>
>> public void set(double[][] value) {
>> this.value = value;
>> }
>>
>> public double[][] getValue() {
>> return value;
>> }
>>
>> @Override
>> public void write(DataOutput out) throws IOException {
>> System.out.println("write");
>> int row=0;
>> int col=0;
>> for(int i=0; i<value.length;i++){
>> row = value.length;
>> for(int j=0; j<value[i].length; j++){
>> col = value[i].length;
>> }
>> }
>> out.writeInt(row);
>> out.writeInt(col);
>>
>> System.out.println("\nTotal no of observations: "+row+":"+col);
>>
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>>
>> out.writeDouble(value[i][j]);
>> }
>> }
>> //priting array
>> for(int vali =0;vali< value.length ;vali ++){
>> for(int valj = 0;valj <value[0].length;valj++){
>> System.out.print(value[vali][valj]+ "\t");
>> }
>> System.out.println("");
>> }
>>
>> }
>>
>> @Override
>> public void readFields(DataInput in) throws IOException {
>> int row = in.readInt();
>> int col = in.readInt();
>>
>> double[][] value = new double[row][col];
>> for(int i=0;i<row ; i++){
>> for(int j= 0 ; j< col;j++){
>> value[i][j] = in.readDouble();
>>
>> }
>> }
>>
>> }
>>
>> @Override
>> public int hashCode() {
>>
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>>
>> }
>>
>>
>> @Override
>> public int compareTo(MatrixWritable o) {
>> // TODO Auto-generated method stub
>> return 0;
>> }
>> @Override
>> public String toString() {
>>
>> return Arrays.toString(value);
>>
>> }
>>
>>
>>
>> }
>>
>> I wrote matrix write,readfields and toString.
>>
>> 1.But my toString is not returning anything. why is it so?
>>
>> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>>
>> public class MyReducer extends
>>
>>
>> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>
>> public void reduce(Iterable<MatrixWritable> key,
>> Iterable<MatrixWritable> values, Context context){
>> for(MatrixWritable c : values){
>>
>> System.out.println("print value "+c.toString());
>>
>> }
>>
>> }
>>
>> but Nothing is printing.when i tried to print value[0].length in toString()
>> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i also needed to print my data asmatrix so i tried
>>
>> public String toString() {
>>
>> String separator = ", ";
>> StringBuffer result = new StringBuffer();
>>
>> // iterate over the first dimension
>> for (int i = 0; i < value.length; i++) {
>> // iterate over the second dimension
>> for(int j = 0; j < value[i].length; j++){
>> result.append(value[i][j]);
>> System.out.print(value[i][j]);
>> result.append(separator);
>> }
>> // remove the last separator
>> result.setLength(result.length() - separator.length());
>> // add a line break.
>> result.append("\n");
>> }
>>
>>
>> return result.toString();
>>
>>
>> }
>>
>>
>>
>> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Can you share the code?
>>>
>>> sent from mobile
>>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> thanks Steve Loughran and Amr Shahin
>>>> Amr Shahin , i refered "
>>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>>> the same thing only. but my toString is not returning anything.
>>>>
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com>wrote:
>>>>
>>>>> Check this out:
>>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>>> It shows how to create a customer writable. If you have "hadoop the
>>>>> definitive guide" there is a really good explanation about custom data
>>>>> types.
>>>>>
>>>>> Happy Halloween
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>>> unmeshabiju@gmail.com>wrote:
>>>>>
>>>>> > this is my post from stackoverflow
>>>>> > but i am not getting any response.
>>>>> >
>>>>> >
>>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>>> are
>>>>> > questions posted in stackoverflow. But they are not answered.
>>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>>> matrix
>>>>> > multiplication in a given dataset.and after that i need to emit the
>>>>> value
>>>>> > of
>>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>>> matrix.so
>>>>> > how to emit these matrices from mapper.And the value should be
>>>>> > corresponding to the key itself.I think for that we need to use
>>>>> > WritableComparable.
>>>>> >
>>>>> >
>>>>> >
>>>>> > public class MatrixWritable implements
>>>>> WritableComparable<MatrixWritable>{
>>>>> >
>>>>> > /**
>>>>> > * @param args
>>>>> > */
>>>>> > private double[][] value;
>>>>> >
>>>>> > public MatrixWritable() {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > set(new double[0][0]);
>>>>> > }
>>>>> >
>>>>> > public MatrixWritable(double[][] value) {
>>>>> > // TODO Auto-generated constructor stub
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public void set(double[][] value) {
>>>>> > this.value = value;
>>>>> > }
>>>>> >
>>>>> > public double[][] getValue() {
>>>>> > return value;
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void write(DataOutput out) throws IOException {
>>>>> > System.out.println("write");
>>>>> > int row=0;
>>>>> > int col=0;
>>>>> > for(int i=0; i<value.length;i++){
>>>>> > row = value.length;
>>>>> > for(int j=0; j<value[i].length; j++){
>>>>> > col = value[i].length;
>>>>> > }
>>>>> > }
>>>>> > out.writeInt(row);
>>>>> > out.writeInt(col);
>>>>> >
>>>>> > System.out.println("\nTotal no of observations:
>>>>> "+row+":"+col);
>>>>> >
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> >
>>>>> > out.writeDouble(value[i][j]);
>>>>> > }
>>>>> > }
>>>>> > //priting array
>>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>>> > System.out.print(value[vali][valj]+ "\t");
>>>>> > }
>>>>> > System.out.println("");
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public void readFields(DataInput in) throws IOException {
>>>>> > int row = in.readInt();
>>>>> > int col = in.readInt();
>>>>> >
>>>>> > double[][] value = new double[row][col];
>>>>> > for(int i=0;i<row ; i++){
>>>>> > for(int j= 0 ; j< col;j++){
>>>>> > value[i][j] = in.readDouble();
>>>>> >
>>>>> > }
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public int hashCode() {
>>>>> >
>>>>> > }
>>>>> >
>>>>> > @Override
>>>>> > public boolean equals(Object o) {
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> > @Override
>>>>> > public int compareTo(MatrixWritable o) {
>>>>> > // TODO Auto-generated method stub
>>>>> > return 0;
>>>>> > }
>>>>> > @Override
>>>>> > public String toString() {
>>>>> >
>>>>> > return Arrays.toString(value);
>>>>> >
>>>>> > }
>>>>> >
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > I wrote matrix write,readfields and toString.
>>>>> >
>>>>> > 1.But my toString is not returning anything. why is it so?
>>>>> >
>>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>>> > emiting only custom value- for checking)
>>>>> >
>>>>> > public class MyReducer extends
>>>>> >
>>>>> >
>>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>>> >
>>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>>> > Iterable<MatrixWritable> values, Context context){
>>>>> > for(MatrixWritable c : values){
>>>>> >
>>>>> > System.out.println("print value "+c.toString());
>>>>> >
>>>>> > }
>>>>> >
>>>>> > }
>>>>> >
>>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>>> toString()
>>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>>> wrong.and i
>>>>> > also needed to print my data asmatrix so i tried
>>>>> >
>>>>> > public String toString() {
>>>>> >
>>>>> > String separator = ", ";
>>>>> > StringBuffer result = new StringBuffer();
>>>>> >
>>>>> > // iterate over the first dimension
>>>>> > for (int i = 0; i < value.length; i++) {
>>>>> > // iterate over the second dimension
>>>>> > for(int j = 0; j < value[i].length; j++){
>>>>> > result.append(value[i][j]);
>>>>> > System.out.print(value[i][j]);
>>>>> > result.append(separator);
>>>>> > }
>>>>> > // remove the last separator
>>>>> > result.setLength(result.length() - separator.length());
>>>>> > // add a line break.
>>>>> > result.append("\n");
>>>>> > }
>>>>> >
>>>>> >
>>>>> > return result.toString();
>>>>> >
>>>>> >
>>>>> > }
>>>>> >
>>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>>> datatype
>>>>> > CompareTo is neccessary right .
>>>>> >
>>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>>> and
>>>>> > what are these methods intended for.
>>>>> > Any Idea.Pls suggest a solution.
>>>>> >
>>>>> > --
>>>>> > *Thanks & Regards*
>>>>> > *
>>>>> > *
>>>>> > Unmesha Sreeveni U.B*
>>>>> > *
>>>>> > *Junior Developer
>>>>> > *
>>>>> > *Amrita Center For Cyber Security
>>>>> > *
>>>>> > *
>>>>> > Amritapuri.
>>>>> >
>>>>> > www.amrita.edu/cyber/
>>>>> > *
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Thanks & Regards*
>>>>
>>>> Unmesha Sreeveni U.B
>>>>
>>>> *Junior Developer*
>>>>
>>>> *Amrita Center For Cyber Security *
>>>>
>>>>
>>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr Shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Can you share the code?
>>
>> sent from mobile
>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>>
>>>
>>> thanks Steve Loughran and Amr Shahin
>>> Amr Shahin , i refered "
>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>> the same thing only. but my toString is not returning anything.
>>>
>>>
>>>
>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Check this out:
>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>> It shows how to create a customer writable. If you have "hadoop the
>>>> definitive guide" there is a really good explanation about custom data
>>>> types.
>>>>
>>>> Happy Halloween
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>> unmeshabiju@gmail.com>wrote:
>>>>
>>>> > this is my post from stackoverflow
>>>> > but i am not getting any response.
>>>> >
>>>> >
>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>> are
>>>> > questions posted in stackoverflow. But they are not answered.
>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>> matrix
>>>> > multiplication in a given dataset.and after that i need to emit the
>>>> value
>>>> > of
>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so
>>>> > how to emit these matrices from mapper.And the value should be
>>>> > corresponding to the key itself.I think for that we need to use
>>>> > WritableComparable.
>>>> >
>>>> >
>>>> >
>>>> > public class MatrixWritable implements
>>>> WritableComparable<MatrixWritable>{
>>>> >
>>>> > /**
>>>> > * @param args
>>>> > */
>>>> > private double[][] value;
>>>> >
>>>> > public MatrixWritable() {
>>>> > // TODO Auto-generated constructor stub
>>>> > set(new double[0][0]);
>>>> > }
>>>> >
>>>> > public MatrixWritable(double[][] value) {
>>>> > // TODO Auto-generated constructor stub
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public void set(double[][] value) {
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public double[][] getValue() {
>>>> > return value;
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void write(DataOutput out) throws IOException {
>>>> > System.out.println("write");
>>>> > int row=0;
>>>> > int col=0;
>>>> > for(int i=0; i<value.length;i++){
>>>> > row = value.length;
>>>> > for(int j=0; j<value[i].length; j++){
>>>> > col = value[i].length;
>>>> > }
>>>> > }
>>>> > out.writeInt(row);
>>>> > out.writeInt(col);
>>>> >
>>>> > System.out.println("\nTotal no of observations:
>>>> "+row+":"+col);
>>>> >
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> >
>>>> > out.writeDouble(value[i][j]);
>>>> > }
>>>> > }
>>>> > //priting array
>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>> > System.out.print(value[vali][valj]+ "\t");
>>>> > }
>>>> > System.out.println("");
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void readFields(DataInput in) throws IOException {
>>>> > int row = in.readInt();
>>>> > int col = in.readInt();
>>>> >
>>>> > double[][] value = new double[row][col];
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> > value[i][j] = in.readDouble();
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public int hashCode() {
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public boolean equals(Object o) {
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > @Override
>>>> > public int compareTo(MatrixWritable o) {
>>>> > // TODO Auto-generated method stub
>>>> > return 0;
>>>> > }
>>>> > @Override
>>>> > public String toString() {
>>>> >
>>>> > return Arrays.toString(value);
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > I wrote matrix write,readfields and toString.
>>>> >
>>>> > 1.But my toString is not returning anything. why is it so?
>>>> >
>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>> > emiting only custom value- for checking)
>>>> >
>>>> > public class MyReducer extends
>>>> >
>>>> >
>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>> >
>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>> > Iterable<MatrixWritable> values, Context context){
>>>> > for(MatrixWritable c : values){
>>>> >
>>>> > System.out.println("print value "+c.toString());
>>>> >
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>> toString()
>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>> wrong.and i
>>>> > also needed to print my data asmatrix so i tried
>>>> >
>>>> > public String toString() {
>>>> >
>>>> > String separator = ", ";
>>>> > StringBuffer result = new StringBuffer();
>>>> >
>>>> > // iterate over the first dimension
>>>> > for (int i = 0; i < value.length; i++) {
>>>> > // iterate over the second dimension
>>>> > for(int j = 0; j < value[i].length; j++){
>>>> > result.append(value[i][j]);
>>>> > System.out.print(value[i][j]);
>>>> > result.append(separator);
>>>> > }
>>>> > // remove the last separator
>>>> > result.setLength(result.length() - separator.length());
>>>> > // add a line break.
>>>> > result.append("\n");
>>>> > }
>>>> >
>>>> >
>>>> > return result.toString();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>> datatype
>>>> > CompareTo is neccessary right .
>>>> >
>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>> and
>>>> > what are these methods intended for.
>>>> > Any Idea.Pls suggest a solution.
>>>> >
>>>> > --
>>>> > *Thanks & Regards*
>>>> > *
>>>> > *
>>>> > Unmesha Sreeveni U.B*
>>>> > *
>>>> > *Junior Developer
>>>> > *
>>>> > *Amrita Center For Cyber Security
>>>> > *
>>>> > *
>>>> > Amritapuri.
>>>> >
>>>> > www.amrita.edu/cyber/
>>>> > *
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr Shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Can you share the code?
>>
>> sent from mobile
>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>>
>>>
>>> thanks Steve Loughran and Amr Shahin
>>> Amr Shahin , i refered "
>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>> the same thing only. but my toString is not returning anything.
>>>
>>>
>>>
>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Check this out:
>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>> It shows how to create a customer writable. If you have "hadoop the
>>>> definitive guide" there is a really good explanation about custom data
>>>> types.
>>>>
>>>> Happy Halloween
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>> unmeshabiju@gmail.com>wrote:
>>>>
>>>> > this is my post from stackoverflow
>>>> > but i am not getting any response.
>>>> >
>>>> >
>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>> are
>>>> > questions posted in stackoverflow. But they are not answered.
>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>> matrix
>>>> > multiplication in a given dataset.and after that i need to emit the
>>>> value
>>>> > of
>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so
>>>> > how to emit these matrices from mapper.And the value should be
>>>> > corresponding to the key itself.I think for that we need to use
>>>> > WritableComparable.
>>>> >
>>>> >
>>>> >
>>>> > public class MatrixWritable implements
>>>> WritableComparable<MatrixWritable>{
>>>> >
>>>> > /**
>>>> > * @param args
>>>> > */
>>>> > private double[][] value;
>>>> >
>>>> > public MatrixWritable() {
>>>> > // TODO Auto-generated constructor stub
>>>> > set(new double[0][0]);
>>>> > }
>>>> >
>>>> > public MatrixWritable(double[][] value) {
>>>> > // TODO Auto-generated constructor stub
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public void set(double[][] value) {
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public double[][] getValue() {
>>>> > return value;
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void write(DataOutput out) throws IOException {
>>>> > System.out.println("write");
>>>> > int row=0;
>>>> > int col=0;
>>>> > for(int i=0; i<value.length;i++){
>>>> > row = value.length;
>>>> > for(int j=0; j<value[i].length; j++){
>>>> > col = value[i].length;
>>>> > }
>>>> > }
>>>> > out.writeInt(row);
>>>> > out.writeInt(col);
>>>> >
>>>> > System.out.println("\nTotal no of observations:
>>>> "+row+":"+col);
>>>> >
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> >
>>>> > out.writeDouble(value[i][j]);
>>>> > }
>>>> > }
>>>> > //priting array
>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>> > System.out.print(value[vali][valj]+ "\t");
>>>> > }
>>>> > System.out.println("");
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void readFields(DataInput in) throws IOException {
>>>> > int row = in.readInt();
>>>> > int col = in.readInt();
>>>> >
>>>> > double[][] value = new double[row][col];
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> > value[i][j] = in.readDouble();
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public int hashCode() {
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public boolean equals(Object o) {
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > @Override
>>>> > public int compareTo(MatrixWritable o) {
>>>> > // TODO Auto-generated method stub
>>>> > return 0;
>>>> > }
>>>> > @Override
>>>> > public String toString() {
>>>> >
>>>> > return Arrays.toString(value);
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > I wrote matrix write,readfields and toString.
>>>> >
>>>> > 1.But my toString is not returning anything. why is it so?
>>>> >
>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>> > emiting only custom value- for checking)
>>>> >
>>>> > public class MyReducer extends
>>>> >
>>>> >
>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>> >
>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>> > Iterable<MatrixWritable> values, Context context){
>>>> > for(MatrixWritable c : values){
>>>> >
>>>> > System.out.println("print value "+c.toString());
>>>> >
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>> toString()
>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>> wrong.and i
>>>> > also needed to print my data asmatrix so i tried
>>>> >
>>>> > public String toString() {
>>>> >
>>>> > String separator = ", ";
>>>> > StringBuffer result = new StringBuffer();
>>>> >
>>>> > // iterate over the first dimension
>>>> > for (int i = 0; i < value.length; i++) {
>>>> > // iterate over the second dimension
>>>> > for(int j = 0; j < value[i].length; j++){
>>>> > result.append(value[i][j]);
>>>> > System.out.print(value[i][j]);
>>>> > result.append(separator);
>>>> > }
>>>> > // remove the last separator
>>>> > result.setLength(result.length() - separator.length());
>>>> > // add a line break.
>>>> > result.append("\n");
>>>> > }
>>>> >
>>>> >
>>>> > return result.toString();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>> datatype
>>>> > CompareTo is neccessary right .
>>>> >
>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>> and
>>>> > what are these methods intended for.
>>>> > Any Idea.Pls suggest a solution.
>>>> >
>>>> > --
>>>> > *Thanks & Regards*
>>>> > *
>>>> > *
>>>> > Unmesha Sreeveni U.B*
>>>> > *
>>>> > *Junior Developer
>>>> > *
>>>> > *Amrita Center For Cyber Security
>>>> > *
>>>> > *
>>>> > Amritapuri.
>>>> >
>>>> > www.amrita.edu/cyber/
>>>> > *
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr Shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Can you share the code?
>>
>> sent from mobile
>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>>
>>>
>>> thanks Steve Loughran and Amr Shahin
>>> Amr Shahin , i refered "
>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>> the same thing only. but my toString is not returning anything.
>>>
>>>
>>>
>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Check this out:
>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>> It shows how to create a customer writable. If you have "hadoop the
>>>> definitive guide" there is a really good explanation about custom data
>>>> types.
>>>>
>>>> Happy Halloween
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>> unmeshabiju@gmail.com>wrote:
>>>>
>>>> > this is my post from stackoverflow
>>>> > but i am not getting any response.
>>>> >
>>>> >
>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>> are
>>>> > questions posted in stackoverflow. But they are not answered.
>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>> matrix
>>>> > multiplication in a given dataset.and after that i need to emit the
>>>> value
>>>> > of
>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so
>>>> > how to emit these matrices from mapper.And the value should be
>>>> > corresponding to the key itself.I think for that we need to use
>>>> > WritableComparable.
>>>> >
>>>> >
>>>> >
>>>> > public class MatrixWritable implements
>>>> WritableComparable<MatrixWritable>{
>>>> >
>>>> > /**
>>>> > * @param args
>>>> > */
>>>> > private double[][] value;
>>>> >
>>>> > public MatrixWritable() {
>>>> > // TODO Auto-generated constructor stub
>>>> > set(new double[0][0]);
>>>> > }
>>>> >
>>>> > public MatrixWritable(double[][] value) {
>>>> > // TODO Auto-generated constructor stub
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public void set(double[][] value) {
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public double[][] getValue() {
>>>> > return value;
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void write(DataOutput out) throws IOException {
>>>> > System.out.println("write");
>>>> > int row=0;
>>>> > int col=0;
>>>> > for(int i=0; i<value.length;i++){
>>>> > row = value.length;
>>>> > for(int j=0; j<value[i].length; j++){
>>>> > col = value[i].length;
>>>> > }
>>>> > }
>>>> > out.writeInt(row);
>>>> > out.writeInt(col);
>>>> >
>>>> > System.out.println("\nTotal no of observations:
>>>> "+row+":"+col);
>>>> >
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> >
>>>> > out.writeDouble(value[i][j]);
>>>> > }
>>>> > }
>>>> > //priting array
>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>> > System.out.print(value[vali][valj]+ "\t");
>>>> > }
>>>> > System.out.println("");
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void readFields(DataInput in) throws IOException {
>>>> > int row = in.readInt();
>>>> > int col = in.readInt();
>>>> >
>>>> > double[][] value = new double[row][col];
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> > value[i][j] = in.readDouble();
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public int hashCode() {
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public boolean equals(Object o) {
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > @Override
>>>> > public int compareTo(MatrixWritable o) {
>>>> > // TODO Auto-generated method stub
>>>> > return 0;
>>>> > }
>>>> > @Override
>>>> > public String toString() {
>>>> >
>>>> > return Arrays.toString(value);
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > I wrote matrix write,readfields and toString.
>>>> >
>>>> > 1.But my toString is not returning anything. why is it so?
>>>> >
>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>> > emiting only custom value- for checking)
>>>> >
>>>> > public class MyReducer extends
>>>> >
>>>> >
>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>> >
>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>> > Iterable<MatrixWritable> values, Context context){
>>>> > for(MatrixWritable c : values){
>>>> >
>>>> > System.out.println("print value "+c.toString());
>>>> >
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>> toString()
>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>> wrong.and i
>>>> > also needed to print my data asmatrix so i tried
>>>> >
>>>> > public String toString() {
>>>> >
>>>> > String separator = ", ";
>>>> > StringBuffer result = new StringBuffer();
>>>> >
>>>> > // iterate over the first dimension
>>>> > for (int i = 0; i < value.length; i++) {
>>>> > // iterate over the second dimension
>>>> > for(int j = 0; j < value[i].length; j++){
>>>> > result.append(value[i][j]);
>>>> > System.out.print(value[i][j]);
>>>> > result.append(separator);
>>>> > }
>>>> > // remove the last separator
>>>> > result.setLength(result.length() - separator.length());
>>>> > // add a line break.
>>>> > result.append("\n");
>>>> > }
>>>> >
>>>> >
>>>> > return result.toString();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>> datatype
>>>> > CompareTo is neccessary right .
>>>> >
>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>> and
>>>> > what are these methods intended for.
>>>> > Any Idea.Pls suggest a solution.
>>>> >
>>>> > --
>>>> > *Thanks & Regards*
>>>> > *
>>>> > *
>>>> > Unmesha Sreeveni U.B*
>>>> > *
>>>> > *Junior Developer
>>>> > *
>>>> > *Amrita Center For Cyber Security
>>>> > *
>>>> > *
>>>> > Amritapuri.
>>>> >
>>>> > www.amrita.edu/cyber/
>>>> > *
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr Shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sun, Nov 3, 2013 at 5:04 PM, unmesha sreeveni <un...@gmail.com>wrote:
> @Amr shahin
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value of
> A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
>
>
> On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Can you share the code?
>>
>> sent from mobile
>> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>>
>>>
>>> thanks Steve Loughran and Amr Shahin
>>> Amr Shahin , i refered "
>>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>>> the same thing only. but my toString is not returning anything.
>>>
>>>
>>>
>>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>>
>>>> Check this out:
>>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>>> It shows how to create a customer writable. If you have "hadoop the
>>>> definitive guide" there is a really good explanation about custom data
>>>> types.
>>>>
>>>> Happy Halloween
>>>>
>>>>
>>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <
>>>> unmeshabiju@gmail.com>wrote:
>>>>
>>>> > this is my post from stackoverflow
>>>> > but i am not getting any response.
>>>> >
>>>> >
>>>> > I need to emit a 2D double array as key and value from mapper.There
>>>> are
>>>> > questions posted in stackoverflow. But they are not answered.
>>>> > we have to create a custom datatype.but how?I am new to these custom
>>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>>> matrix
>>>> > multiplication in a given dataset.and after that i need to emit the
>>>> value
>>>> > of
>>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>>> matrix.so
>>>> > how to emit these matrices from mapper.And the value should be
>>>> > corresponding to the key itself.I think for that we need to use
>>>> > WritableComparable.
>>>> >
>>>> >
>>>> >
>>>> > public class MatrixWritable implements
>>>> WritableComparable<MatrixWritable>{
>>>> >
>>>> > /**
>>>> > * @param args
>>>> > */
>>>> > private double[][] value;
>>>> >
>>>> > public MatrixWritable() {
>>>> > // TODO Auto-generated constructor stub
>>>> > set(new double[0][0]);
>>>> > }
>>>> >
>>>> > public MatrixWritable(double[][] value) {
>>>> > // TODO Auto-generated constructor stub
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public void set(double[][] value) {
>>>> > this.value = value;
>>>> > }
>>>> >
>>>> > public double[][] getValue() {
>>>> > return value;
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void write(DataOutput out) throws IOException {
>>>> > System.out.println("write");
>>>> > int row=0;
>>>> > int col=0;
>>>> > for(int i=0; i<value.length;i++){
>>>> > row = value.length;
>>>> > for(int j=0; j<value[i].length; j++){
>>>> > col = value[i].length;
>>>> > }
>>>> > }
>>>> > out.writeInt(row);
>>>> > out.writeInt(col);
>>>> >
>>>> > System.out.println("\nTotal no of observations:
>>>> "+row+":"+col);
>>>> >
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> >
>>>> > out.writeDouble(value[i][j]);
>>>> > }
>>>> > }
>>>> > //priting array
>>>> > for(int vali =0;vali< value.length ;vali ++){
>>>> > for(int valj = 0;valj <value[0].length;valj++){
>>>> > System.out.print(value[vali][valj]+ "\t");
>>>> > }
>>>> > System.out.println("");
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public void readFields(DataInput in) throws IOException {
>>>> > int row = in.readInt();
>>>> > int col = in.readInt();
>>>> >
>>>> > double[][] value = new double[row][col];
>>>> > for(int i=0;i<row ; i++){
>>>> > for(int j= 0 ; j< col;j++){
>>>> > value[i][j] = in.readDouble();
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public int hashCode() {
>>>> >
>>>> > }
>>>> >
>>>> > @Override
>>>> > public boolean equals(Object o) {
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > @Override
>>>> > public int compareTo(MatrixWritable o) {
>>>> > // TODO Auto-generated method stub
>>>> > return 0;
>>>> > }
>>>> > @Override
>>>> > public String toString() {
>>>> >
>>>> > return Arrays.toString(value);
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > I wrote matrix write,readfields and toString.
>>>> >
>>>> > 1.But my toString is not returning anything. why is it so?
>>>> >
>>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>>> > emiting only custom value- for checking)
>>>> >
>>>> > public class MyReducer extends
>>>> >
>>>> >
>>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>>> >
>>>> > public void reduce(Iterable<MatrixWritable> key,
>>>> > Iterable<MatrixWritable> values, Context context){
>>>> > for(MatrixWritable c : values){
>>>> >
>>>> > System.out.println("print value "+c.toString());
>>>> >
>>>> > }
>>>> >
>>>> > }
>>>> >
>>>> > but Nothing is printing.when i tried to print value[0].length in
>>>> toString()
>>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>>> wrong.and i
>>>> > also needed to print my data asmatrix so i tried
>>>> >
>>>> > public String toString() {
>>>> >
>>>> > String separator = ", ";
>>>> > StringBuffer result = new StringBuffer();
>>>> >
>>>> > // iterate over the first dimension
>>>> > for (int i = 0; i < value.length; i++) {
>>>> > // iterate over the second dimension
>>>> > for(int j = 0; j < value[i].length; j++){
>>>> > result.append(value[i][j]);
>>>> > System.out.print(value[i][j]);
>>>> > result.append(separator);
>>>> > }
>>>> > // remove the last separator
>>>> > result.setLength(result.length() - separator.length());
>>>> > // add a line break.
>>>> > result.append("\n");
>>>> > }
>>>> >
>>>> >
>>>> > return result.toString();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>>> datatype
>>>> > CompareTo is neccessary right .
>>>> >
>>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>>> and
>>>> > what are these methods intended for.
>>>> > Any Idea.Pls suggest a solution.
>>>> >
>>>> > --
>>>> > *Thanks & Regards*
>>>> > *
>>>> > *
>>>> > Unmesha Sreeveni U.B*
>>>> > *
>>>> > *Junior Developer
>>>> > *
>>>> > *Amrita Center For Cyber Security
>>>> > *
>>>> > *
>>>> > Amritapuri.
>>>> >
>>>> > www.amrita.edu/cyber/
>>>> > *
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>> *Amrita Center For Cyber Security *
>>>
>>>
>>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>>
>>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
> Can you share the code?
>
> sent from mobile
> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
>>
>> thanks Steve Loughran and Amr Shahin
>> Amr Shahin , i refered "
>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>> the same thing only. but my toString is not returning anything.
>>
>>
>>
>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Check this out:
>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>> It shows how to create a customer writable. If you have "hadoop the
>>> definitive guide" there is a really good explanation about custom data
>>> types.
>>>
>>> Happy Halloween
>>>
>>>
>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>> >wrote:
>>>
>>> > this is my post from stackoverflow
>>> > but i am not getting any response.
>>> >
>>> >
>>> > I need to emit a 2D double array as key and value from mapper.There are
>>> > questions posted in stackoverflow. But they are not answered.
>>> > we have to create a custom datatype.but how?I am new to these custom
>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>> matrix
>>> > multiplication in a given dataset.and after that i need to emit the
>>> value
>>> > of
>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so
>>> > how to emit these matrices from mapper.And the value should be
>>> > corresponding to the key itself.I think for that we need to use
>>> > WritableComparable.
>>> >
>>> >
>>> >
>>> > public class MatrixWritable implements
>>> WritableComparable<MatrixWritable>{
>>> >
>>> > /**
>>> > * @param args
>>> > */
>>> > private double[][] value;
>>> >
>>> > public MatrixWritable() {
>>> > // TODO Auto-generated constructor stub
>>> > set(new double[0][0]);
>>> > }
>>> >
>>> > public MatrixWritable(double[][] value) {
>>> > // TODO Auto-generated constructor stub
>>> > this.value = value;
>>> > }
>>> >
>>> > public void set(double[][] value) {
>>> > this.value = value;
>>> > }
>>> >
>>> > public double[][] getValue() {
>>> > return value;
>>> > }
>>> >
>>> > @Override
>>> > public void write(DataOutput out) throws IOException {
>>> > System.out.println("write");
>>> > int row=0;
>>> > int col=0;
>>> > for(int i=0; i<value.length;i++){
>>> > row = value.length;
>>> > for(int j=0; j<value[i].length; j++){
>>> > col = value[i].length;
>>> > }
>>> > }
>>> > out.writeInt(row);
>>> > out.writeInt(col);
>>> >
>>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>>> >
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> >
>>> > out.writeDouble(value[i][j]);
>>> > }
>>> > }
>>> > //priting array
>>> > for(int vali =0;vali< value.length ;vali ++){
>>> > for(int valj = 0;valj <value[0].length;valj++){
>>> > System.out.print(value[vali][valj]+ "\t");
>>> > }
>>> > System.out.println("");
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public void readFields(DataInput in) throws IOException {
>>> > int row = in.readInt();
>>> > int col = in.readInt();
>>> >
>>> > double[][] value = new double[row][col];
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> > value[i][j] = in.readDouble();
>>> >
>>> > }
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public int hashCode() {
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public boolean equals(Object o) {
>>> >
>>> > }
>>> >
>>> >
>>> > @Override
>>> > public int compareTo(MatrixWritable o) {
>>> > // TODO Auto-generated method stub
>>> > return 0;
>>> > }
>>> > @Override
>>> > public String toString() {
>>> >
>>> > return Arrays.toString(value);
>>> >
>>> > }
>>> >
>>> >
>>> >
>>> > }
>>> >
>>> > I wrote matrix write,readfields and toString.
>>> >
>>> > 1.But my toString is not returning anything. why is it so?
>>> >
>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>> > emiting only custom value- for checking)
>>> >
>>> > public class MyReducer extends
>>> >
>>> >
>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>> >
>>> > public void reduce(Iterable<MatrixWritable> key,
>>> > Iterable<MatrixWritable> values, Context context){
>>> > for(MatrixWritable c : values){
>>> >
>>> > System.out.println("print value "+c.toString());
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> > but Nothing is printing.when i tried to print value[0].length in
>>> toString()
>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>> wrong.and i
>>> > also needed to print my data asmatrix so i tried
>>> >
>>> > public String toString() {
>>> >
>>> > String separator = ", ";
>>> > StringBuffer result = new StringBuffer();
>>> >
>>> > // iterate over the first dimension
>>> > for (int i = 0; i < value.length; i++) {
>>> > // iterate over the second dimension
>>> > for(int j = 0; j < value[i].length; j++){
>>> > result.append(value[i][j]);
>>> > System.out.print(value[i][j]);
>>> > result.append(separator);
>>> > }
>>> > // remove the last separator
>>> > result.setLength(result.length() - separator.length());
>>> > // add a line break.
>>> > result.append("\n");
>>> > }
>>> >
>>> >
>>> > return result.toString();
>>> >
>>> >
>>> > }
>>> >
>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>> datatype
>>> > CompareTo is neccessary right .
>>> >
>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>> and
>>> > what are these methods intended for.
>>> > Any Idea.Pls suggest a solution.
>>> >
>>> > --
>>> > *Thanks & Regards*
>>> > *
>>> > *
>>> > Unmesha Sreeveni U.B*
>>> > *
>>> > *Junior Developer
>>> > *
>>> > *Amrita Center For Cyber Security
>>> > *
>>> > *
>>> > Amritapuri.
>>> >
>>> > www.amrita.edu/cyber/
>>> > *
>>> >
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
> Can you share the code?
>
> sent from mobile
> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
>>
>> thanks Steve Loughran and Amr Shahin
>> Amr Shahin , i refered "
>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>> the same thing only. but my toString is not returning anything.
>>
>>
>>
>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Check this out:
>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>> It shows how to create a customer writable. If you have "hadoop the
>>> definitive guide" there is a really good explanation about custom data
>>> types.
>>>
>>> Happy Halloween
>>>
>>>
>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>> >wrote:
>>>
>>> > this is my post from stackoverflow
>>> > but i am not getting any response.
>>> >
>>> >
>>> > I need to emit a 2D double array as key and value from mapper.There are
>>> > questions posted in stackoverflow. But they are not answered.
>>> > we have to create a custom datatype.but how?I am new to these custom
>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>> matrix
>>> > multiplication in a given dataset.and after that i need to emit the
>>> value
>>> > of
>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so
>>> > how to emit these matrices from mapper.And the value should be
>>> > corresponding to the key itself.I think for that we need to use
>>> > WritableComparable.
>>> >
>>> >
>>> >
>>> > public class MatrixWritable implements
>>> WritableComparable<MatrixWritable>{
>>> >
>>> > /**
>>> > * @param args
>>> > */
>>> > private double[][] value;
>>> >
>>> > public MatrixWritable() {
>>> > // TODO Auto-generated constructor stub
>>> > set(new double[0][0]);
>>> > }
>>> >
>>> > public MatrixWritable(double[][] value) {
>>> > // TODO Auto-generated constructor stub
>>> > this.value = value;
>>> > }
>>> >
>>> > public void set(double[][] value) {
>>> > this.value = value;
>>> > }
>>> >
>>> > public double[][] getValue() {
>>> > return value;
>>> > }
>>> >
>>> > @Override
>>> > public void write(DataOutput out) throws IOException {
>>> > System.out.println("write");
>>> > int row=0;
>>> > int col=0;
>>> > for(int i=0; i<value.length;i++){
>>> > row = value.length;
>>> > for(int j=0; j<value[i].length; j++){
>>> > col = value[i].length;
>>> > }
>>> > }
>>> > out.writeInt(row);
>>> > out.writeInt(col);
>>> >
>>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>>> >
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> >
>>> > out.writeDouble(value[i][j]);
>>> > }
>>> > }
>>> > //priting array
>>> > for(int vali =0;vali< value.length ;vali ++){
>>> > for(int valj = 0;valj <value[0].length;valj++){
>>> > System.out.print(value[vali][valj]+ "\t");
>>> > }
>>> > System.out.println("");
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public void readFields(DataInput in) throws IOException {
>>> > int row = in.readInt();
>>> > int col = in.readInt();
>>> >
>>> > double[][] value = new double[row][col];
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> > value[i][j] = in.readDouble();
>>> >
>>> > }
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public int hashCode() {
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public boolean equals(Object o) {
>>> >
>>> > }
>>> >
>>> >
>>> > @Override
>>> > public int compareTo(MatrixWritable o) {
>>> > // TODO Auto-generated method stub
>>> > return 0;
>>> > }
>>> > @Override
>>> > public String toString() {
>>> >
>>> > return Arrays.toString(value);
>>> >
>>> > }
>>> >
>>> >
>>> >
>>> > }
>>> >
>>> > I wrote matrix write,readfields and toString.
>>> >
>>> > 1.But my toString is not returning anything. why is it so?
>>> >
>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>> > emiting only custom value- for checking)
>>> >
>>> > public class MyReducer extends
>>> >
>>> >
>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>> >
>>> > public void reduce(Iterable<MatrixWritable> key,
>>> > Iterable<MatrixWritable> values, Context context){
>>> > for(MatrixWritable c : values){
>>> >
>>> > System.out.println("print value "+c.toString());
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> > but Nothing is printing.when i tried to print value[0].length in
>>> toString()
>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>> wrong.and i
>>> > also needed to print my data asmatrix so i tried
>>> >
>>> > public String toString() {
>>> >
>>> > String separator = ", ";
>>> > StringBuffer result = new StringBuffer();
>>> >
>>> > // iterate over the first dimension
>>> > for (int i = 0; i < value.length; i++) {
>>> > // iterate over the second dimension
>>> > for(int j = 0; j < value[i].length; j++){
>>> > result.append(value[i][j]);
>>> > System.out.print(value[i][j]);
>>> > result.append(separator);
>>> > }
>>> > // remove the last separator
>>> > result.setLength(result.length() - separator.length());
>>> > // add a line break.
>>> > result.append("\n");
>>> > }
>>> >
>>> >
>>> > return result.toString();
>>> >
>>> >
>>> > }
>>> >
>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>> datatype
>>> > CompareTo is neccessary right .
>>> >
>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>> and
>>> > what are these methods intended for.
>>> > Any Idea.Pls suggest a solution.
>>> >
>>> > --
>>> > *Thanks & Regards*
>>> > *
>>> > *
>>> > Unmesha Sreeveni U.B*
>>> > *
>>> > *Junior Developer
>>> > *
>>> > *Amrita Center For Cyber Security
>>> > *
>>> > *
>>> > Amritapuri.
>>> >
>>> > www.amrita.edu/cyber/
>>> > *
>>> >
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
> Can you share the code?
>
> sent from mobile
> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
>>
>> thanks Steve Loughran and Amr Shahin
>> Amr Shahin , i refered "
>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>> the same thing only. but my toString is not returning anything.
>>
>>
>>
>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Check this out:
>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>> It shows how to create a customer writable. If you have "hadoop the
>>> definitive guide" there is a really good explanation about custom data
>>> types.
>>>
>>> Happy Halloween
>>>
>>>
>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>> >wrote:
>>>
>>> > this is my post from stackoverflow
>>> > but i am not getting any response.
>>> >
>>> >
>>> > I need to emit a 2D double array as key and value from mapper.There are
>>> > questions posted in stackoverflow. But they are not answered.
>>> > we have to create a custom datatype.but how?I am new to these custom
>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>> matrix
>>> > multiplication in a given dataset.and after that i need to emit the
>>> value
>>> > of
>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so
>>> > how to emit these matrices from mapper.And the value should be
>>> > corresponding to the key itself.I think for that we need to use
>>> > WritableComparable.
>>> >
>>> >
>>> >
>>> > public class MatrixWritable implements
>>> WritableComparable<MatrixWritable>{
>>> >
>>> > /**
>>> > * @param args
>>> > */
>>> > private double[][] value;
>>> >
>>> > public MatrixWritable() {
>>> > // TODO Auto-generated constructor stub
>>> > set(new double[0][0]);
>>> > }
>>> >
>>> > public MatrixWritable(double[][] value) {
>>> > // TODO Auto-generated constructor stub
>>> > this.value = value;
>>> > }
>>> >
>>> > public void set(double[][] value) {
>>> > this.value = value;
>>> > }
>>> >
>>> > public double[][] getValue() {
>>> > return value;
>>> > }
>>> >
>>> > @Override
>>> > public void write(DataOutput out) throws IOException {
>>> > System.out.println("write");
>>> > int row=0;
>>> > int col=0;
>>> > for(int i=0; i<value.length;i++){
>>> > row = value.length;
>>> > for(int j=0; j<value[i].length; j++){
>>> > col = value[i].length;
>>> > }
>>> > }
>>> > out.writeInt(row);
>>> > out.writeInt(col);
>>> >
>>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>>> >
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> >
>>> > out.writeDouble(value[i][j]);
>>> > }
>>> > }
>>> > //priting array
>>> > for(int vali =0;vali< value.length ;vali ++){
>>> > for(int valj = 0;valj <value[0].length;valj++){
>>> > System.out.print(value[vali][valj]+ "\t");
>>> > }
>>> > System.out.println("");
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public void readFields(DataInput in) throws IOException {
>>> > int row = in.readInt();
>>> > int col = in.readInt();
>>> >
>>> > double[][] value = new double[row][col];
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> > value[i][j] = in.readDouble();
>>> >
>>> > }
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public int hashCode() {
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public boolean equals(Object o) {
>>> >
>>> > }
>>> >
>>> >
>>> > @Override
>>> > public int compareTo(MatrixWritable o) {
>>> > // TODO Auto-generated method stub
>>> > return 0;
>>> > }
>>> > @Override
>>> > public String toString() {
>>> >
>>> > return Arrays.toString(value);
>>> >
>>> > }
>>> >
>>> >
>>> >
>>> > }
>>> >
>>> > I wrote matrix write,readfields and toString.
>>> >
>>> > 1.But my toString is not returning anything. why is it so?
>>> >
>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>> > emiting only custom value- for checking)
>>> >
>>> > public class MyReducer extends
>>> >
>>> >
>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>> >
>>> > public void reduce(Iterable<MatrixWritable> key,
>>> > Iterable<MatrixWritable> values, Context context){
>>> > for(MatrixWritable c : values){
>>> >
>>> > System.out.println("print value "+c.toString());
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> > but Nothing is printing.when i tried to print value[0].length in
>>> toString()
>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>> wrong.and i
>>> > also needed to print my data asmatrix so i tried
>>> >
>>> > public String toString() {
>>> >
>>> > String separator = ", ";
>>> > StringBuffer result = new StringBuffer();
>>> >
>>> > // iterate over the first dimension
>>> > for (int i = 0; i < value.length; i++) {
>>> > // iterate over the second dimension
>>> > for(int j = 0; j < value[i].length; j++){
>>> > result.append(value[i][j]);
>>> > System.out.print(value[i][j]);
>>> > result.append(separator);
>>> > }
>>> > // remove the last separator
>>> > result.setLength(result.length() - separator.length());
>>> > // add a line break.
>>> > result.append("\n");
>>> > }
>>> >
>>> >
>>> > return result.toString();
>>> >
>>> >
>>> > }
>>> >
>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>> datatype
>>> > CompareTo is neccessary right .
>>> >
>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>> and
>>> > what are these methods intended for.
>>> > Any Idea.Pls suggest a solution.
>>> >
>>> > --
>>> > *Thanks & Regards*
>>> > *
>>> > *
>>> > Unmesha Sreeveni U.B*
>>> > *
>>> > *Junior Developer
>>> > *
>>> > *Amrita Center For Cyber Security
>>> > *
>>> > *
>>> > Amritapuri.
>>> >
>>> > www.amrita.edu/cyber/
>>> > *
>>> >
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
@Amr shahin
this is my post from stackoverflow
but i am not getting any response.
I need to emit a 2D double array as key and value from mapper.There are
questions posted in stackoverflow. But they are not answered.
we have to create a custom datatype.but how?I am new to these custom
datatypes. i dnt have any idea where to start.I am doing some of the matrix
multiplication in a given dataset.and after that i need to emit the value of
A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
how to emit these matrices from mapper.And the value should be
corresponding to the key itself.I think for that we need to use
WritableComparable.
public class MatrixWritable implements WritableComparable<MatrixWritable>{
/**
* @param args
*/
private double[][] value;
public MatrixWritable() {
// TODO Auto-generated constructor stub
set(new double[0][0]);
}
public MatrixWritable(double[][] value) {
// TODO Auto-generated constructor stub
this.value = value;
}
public void set(double[][] value) {
this.value = value;
}
public double[][] getValue() {
return value;
}
@Override
public void write(DataOutput out) throws IOException {
System.out.println("write");
int row=0;
int col=0;
for(int i=0; i<value.length;i++){
row = value.length;
for(int j=0; j<value[i].length; j++){
col = value[i].length;
}
}
out.writeInt(row);
out.writeInt(col);
System.out.println("\nTotal no of observations: "+row+":"+col);
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
out.writeDouble(value[i][j]);
}
}
//priting array
for(int vali =0;vali< value.length ;vali ++){
for(int valj = 0;valj <value[0].length;valj++){
System.out.print(value[vali][valj]+ "\t");
}
System.out.println("");
}
}
@Override
public void readFields(DataInput in) throws IOException {
int row = in.readInt();
int col = in.readInt();
double[][] value = new double[row][col];
for(int i=0;i<row ; i++){
for(int j= 0 ; j< col;j++){
value[i][j] = in.readDouble();
}
}
}
@Override
public int hashCode() {
}
@Override
public boolean equals(Object o) {
}
@Override
public int compareTo(MatrixWritable o) {
// TODO Auto-generated method stub
return 0;
}
@Override
public String toString() {
return Arrays.toString(value);
}
}
I wrote matrix write,readfields and toString.
1.But my toString is not returning anything. why is it so?
2.How to print these values with in Reducer ?I tried doing(tried with
emiting only custom value- for checking)
public class MyReducer extends
Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
public void reduce(Iterable<MatrixWritable> key,
Iterable<MatrixWritable> values, Context context){
for(MatrixWritable c : values){
System.out.println("print value "+c.toString());
}
}
but Nothing is printing.when i tried to print value[0].length in toString()
method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
also needed to print my data asmatrix so i tried
public String toString() {
String separator = ", ";
StringBuffer result = new StringBuffer();
// iterate over the first dimension
for (int i = 0; i < value.length; i++) {
// iterate over the second dimension
for(int j = 0; j < value[i].length; j++){
result.append(value[i][j]);
System.out.print(value[i][j]);
result.append(separator);
}
// remove the last separator
result.setLength(result.length() - separator.length());
// add a line break.
result.append("\n");
}
return result.toString();
}
On Sat, Nov 2, 2013 at 7:56 PM, Amr Shahin <am...@gmail.com> wrote:
> Can you share the code?
>
> sent from mobile
> On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
>>
>> thanks Steve Loughran and Amr Shahin
>> Amr Shahin , i refered "
>> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
>> the same thing only. but my toString is not returning anything.
>>
>>
>>
>> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>>
>>> Check this out:
>>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>>> It shows how to create a customer writable. If you have "hadoop the
>>> definitive guide" there is a really good explanation about custom data
>>> types.
>>>
>>> Happy Halloween
>>>
>>>
>>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>>> >wrote:
>>>
>>> > this is my post from stackoverflow
>>> > but i am not getting any response.
>>> >
>>> >
>>> > I need to emit a 2D double array as key and value from mapper.There are
>>> > questions posted in stackoverflow. But they are not answered.
>>> > we have to create a custom datatype.but how?I am new to these custom
>>> > datatypes. i dnt have any idea where to start.I am doing some of the
>>> matrix
>>> > multiplication in a given dataset.and after that i need to emit the
>>> value
>>> > of
>>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>>> matrix.so
>>> > how to emit these matrices from mapper.And the value should be
>>> > corresponding to the key itself.I think for that we need to use
>>> > WritableComparable.
>>> >
>>> >
>>> >
>>> > public class MatrixWritable implements
>>> WritableComparable<MatrixWritable>{
>>> >
>>> > /**
>>> > * @param args
>>> > */
>>> > private double[][] value;
>>> >
>>> > public MatrixWritable() {
>>> > // TODO Auto-generated constructor stub
>>> > set(new double[0][0]);
>>> > }
>>> >
>>> > public MatrixWritable(double[][] value) {
>>> > // TODO Auto-generated constructor stub
>>> > this.value = value;
>>> > }
>>> >
>>> > public void set(double[][] value) {
>>> > this.value = value;
>>> > }
>>> >
>>> > public double[][] getValue() {
>>> > return value;
>>> > }
>>> >
>>> > @Override
>>> > public void write(DataOutput out) throws IOException {
>>> > System.out.println("write");
>>> > int row=0;
>>> > int col=0;
>>> > for(int i=0; i<value.length;i++){
>>> > row = value.length;
>>> > for(int j=0; j<value[i].length; j++){
>>> > col = value[i].length;
>>> > }
>>> > }
>>> > out.writeInt(row);
>>> > out.writeInt(col);
>>> >
>>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>>> >
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> >
>>> > out.writeDouble(value[i][j]);
>>> > }
>>> > }
>>> > //priting array
>>> > for(int vali =0;vali< value.length ;vali ++){
>>> > for(int valj = 0;valj <value[0].length;valj++){
>>> > System.out.print(value[vali][valj]+ "\t");
>>> > }
>>> > System.out.println("");
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public void readFields(DataInput in) throws IOException {
>>> > int row = in.readInt();
>>> > int col = in.readInt();
>>> >
>>> > double[][] value = new double[row][col];
>>> > for(int i=0;i<row ; i++){
>>> > for(int j= 0 ; j< col;j++){
>>> > value[i][j] = in.readDouble();
>>> >
>>> > }
>>> > }
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public int hashCode() {
>>> >
>>> > }
>>> >
>>> > @Override
>>> > public boolean equals(Object o) {
>>> >
>>> > }
>>> >
>>> >
>>> > @Override
>>> > public int compareTo(MatrixWritable o) {
>>> > // TODO Auto-generated method stub
>>> > return 0;
>>> > }
>>> > @Override
>>> > public String toString() {
>>> >
>>> > return Arrays.toString(value);
>>> >
>>> > }
>>> >
>>> >
>>> >
>>> > }
>>> >
>>> > I wrote matrix write,readfields and toString.
>>> >
>>> > 1.But my toString is not returning anything. why is it so?
>>> >
>>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>>> > emiting only custom value- for checking)
>>> >
>>> > public class MyReducer extends
>>> >
>>> >
>>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>>> >
>>> > public void reduce(Iterable<MatrixWritable> key,
>>> > Iterable<MatrixWritable> values, Context context){
>>> > for(MatrixWritable c : values){
>>> >
>>> > System.out.println("print value "+c.toString());
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> > but Nothing is printing.when i tried to print value[0].length in
>>> toString()
>>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing
>>> wrong.and i
>>> > also needed to print my data asmatrix so i tried
>>> >
>>> > public String toString() {
>>> >
>>> > String separator = ", ";
>>> > StringBuffer result = new StringBuffer();
>>> >
>>> > // iterate over the first dimension
>>> > for (int i = 0; i < value.length; i++) {
>>> > // iterate over the second dimension
>>> > for(int j = 0; j < value[i].length; j++){
>>> > result.append(value[i][j]);
>>> > System.out.print(value[i][j]);
>>> > result.append(separator);
>>> > }
>>> > // remove the last separator
>>> > result.setLength(result.length() - separator.length());
>>> > // add a line break.
>>> > result.append("\n");
>>> > }
>>> >
>>> >
>>> > return result.toString();
>>> >
>>> >
>>> > }
>>> >
>>> > Again my output is empty. 3.Inorder to emit a key too as custom
>>> datatype
>>> > CompareTo is neccessary right .
>>> >
>>> > 4.so what should i include in that methods CompareTo,hashcode,equals
>>> and
>>> > what are these methods intended for.
>>> > Any Idea.Pls suggest a solution.
>>> >
>>> > --
>>> > *Thanks & Regards*
>>> > *
>>> > *
>>> > Unmesha Sreeveni U.B*
>>> > *
>>> > *Junior Developer
>>> > *
>>> > *Amrita Center For Cyber Security
>>> > *
>>> > *
>>> > Amritapuri.
>>> >
>>> > www.amrita.edu/cyber/
>>> > *
>>> >
>>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>> *Amrita Center For Cyber Security *
>>
>>
>> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>>
>
--
*Thanks & Regards*
Unmesha Sreeveni U.B
*Junior Developer*
*Amrita Center For Cyber Security*
* Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Can you share the code?
sent from mobile
On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
> thanks Steve Loughran and Amr Shahin
> Amr Shahin , i refered "
> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
> the same thing only. but my toString is not returning anything.
>
>
>
> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Check this out:
>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>> It shows how to create a customer writable. If you have "hadoop the
>> definitive guide" there is a really good explanation about custom data
>> types.
>>
>> Happy Halloween
>>
>>
>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>> >wrote:
>>
>> > this is my post from stackoverflow
>> > but i am not getting any response.
>> >
>> >
>> > I need to emit a 2D double array as key and value from mapper.There are
>> > questions posted in stackoverflow. But they are not answered.
>> > we have to create a custom datatype.but how?I am new to these custom
>> > datatypes. i dnt have any idea where to start.I am doing some of the
>> matrix
>> > multiplication in a given dataset.and after that i need to emit the
>> value
>> > of
>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so
>> > how to emit these matrices from mapper.And the value should be
>> > corresponding to the key itself.I think for that we need to use
>> > WritableComparable.
>> >
>> >
>> >
>> > public class MatrixWritable implements
>> WritableComparable<MatrixWritable>{
>> >
>> > /**
>> > * @param args
>> > */
>> > private double[][] value;
>> >
>> > public MatrixWritable() {
>> > // TODO Auto-generated constructor stub
>> > set(new double[0][0]);
>> > }
>> >
>> > public MatrixWritable(double[][] value) {
>> > // TODO Auto-generated constructor stub
>> > this.value = value;
>> > }
>> >
>> > public void set(double[][] value) {
>> > this.value = value;
>> > }
>> >
>> > public double[][] getValue() {
>> > return value;
>> > }
>> >
>> > @Override
>> > public void write(DataOutput out) throws IOException {
>> > System.out.println("write");
>> > int row=0;
>> > int col=0;
>> > for(int i=0; i<value.length;i++){
>> > row = value.length;
>> > for(int j=0; j<value[i].length; j++){
>> > col = value[i].length;
>> > }
>> > }
>> > out.writeInt(row);
>> > out.writeInt(col);
>> >
>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>> >
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> >
>> > out.writeDouble(value[i][j]);
>> > }
>> > }
>> > //priting array
>> > for(int vali =0;vali< value.length ;vali ++){
>> > for(int valj = 0;valj <value[0].length;valj++){
>> > System.out.print(value[vali][valj]+ "\t");
>> > }
>> > System.out.println("");
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public void readFields(DataInput in) throws IOException {
>> > int row = in.readInt();
>> > int col = in.readInt();
>> >
>> > double[][] value = new double[row][col];
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> > value[i][j] = in.readDouble();
>> >
>> > }
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public int hashCode() {
>> >
>> > }
>> >
>> > @Override
>> > public boolean equals(Object o) {
>> >
>> > }
>> >
>> >
>> > @Override
>> > public int compareTo(MatrixWritable o) {
>> > // TODO Auto-generated method stub
>> > return 0;
>> > }
>> > @Override
>> > public String toString() {
>> >
>> > return Arrays.toString(value);
>> >
>> > }
>> >
>> >
>> >
>> > }
>> >
>> > I wrote matrix write,readfields and toString.
>> >
>> > 1.But my toString is not returning anything. why is it so?
>> >
>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>> > emiting only custom value- for checking)
>> >
>> > public class MyReducer extends
>> >
>> >
>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>> >
>> > public void reduce(Iterable<MatrixWritable> key,
>> > Iterable<MatrixWritable> values, Context context){
>> > for(MatrixWritable c : values){
>> >
>> > System.out.println("print value "+c.toString());
>> >
>> > }
>> >
>> > }
>> >
>> > but Nothing is printing.when i tried to print value[0].length in
>> toString()
>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i
>> > also needed to print my data asmatrix so i tried
>> >
>> > public String toString() {
>> >
>> > String separator = ", ";
>> > StringBuffer result = new StringBuffer();
>> >
>> > // iterate over the first dimension
>> > for (int i = 0; i < value.length; i++) {
>> > // iterate over the second dimension
>> > for(int j = 0; j < value[i].length; j++){
>> > result.append(value[i][j]);
>> > System.out.print(value[i][j]);
>> > result.append(separator);
>> > }
>> > // remove the last separator
>> > result.setLength(result.length() - separator.length());
>> > // add a line break.
>> > result.append("\n");
>> > }
>> >
>> >
>> > return result.toString();
>> >
>> >
>> > }
>> >
>> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
>> > CompareTo is neccessary right .
>> >
>> > 4.so what should i include in that methods CompareTo,hashcode,equals and
>> > what are these methods intended for.
>> > Any Idea.Pls suggest a solution.
>> >
>> > --
>> > *Thanks & Regards*
>> > *
>> > *
>> > Unmesha Sreeveni U.B*
>> > *
>> > *Junior Developer
>> > *
>> > *Amrita Center For Cyber Security
>> > *
>> > *
>> > Amritapuri.
>> >
>> > www.amrita.edu/cyber/
>> > *
>> >
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Can you share the code?
sent from mobile
On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
> thanks Steve Loughran and Amr Shahin
> Amr Shahin , i refered "
> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
> the same thing only. but my toString is not returning anything.
>
>
>
> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Check this out:
>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>> It shows how to create a customer writable. If you have "hadoop the
>> definitive guide" there is a really good explanation about custom data
>> types.
>>
>> Happy Halloween
>>
>>
>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>> >wrote:
>>
>> > this is my post from stackoverflow
>> > but i am not getting any response.
>> >
>> >
>> > I need to emit a 2D double array as key and value from mapper.There are
>> > questions posted in stackoverflow. But they are not answered.
>> > we have to create a custom datatype.but how?I am new to these custom
>> > datatypes. i dnt have any idea where to start.I am doing some of the
>> matrix
>> > multiplication in a given dataset.and after that i need to emit the
>> value
>> > of
>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so
>> > how to emit these matrices from mapper.And the value should be
>> > corresponding to the key itself.I think for that we need to use
>> > WritableComparable.
>> >
>> >
>> >
>> > public class MatrixWritable implements
>> WritableComparable<MatrixWritable>{
>> >
>> > /**
>> > * @param args
>> > */
>> > private double[][] value;
>> >
>> > public MatrixWritable() {
>> > // TODO Auto-generated constructor stub
>> > set(new double[0][0]);
>> > }
>> >
>> > public MatrixWritable(double[][] value) {
>> > // TODO Auto-generated constructor stub
>> > this.value = value;
>> > }
>> >
>> > public void set(double[][] value) {
>> > this.value = value;
>> > }
>> >
>> > public double[][] getValue() {
>> > return value;
>> > }
>> >
>> > @Override
>> > public void write(DataOutput out) throws IOException {
>> > System.out.println("write");
>> > int row=0;
>> > int col=0;
>> > for(int i=0; i<value.length;i++){
>> > row = value.length;
>> > for(int j=0; j<value[i].length; j++){
>> > col = value[i].length;
>> > }
>> > }
>> > out.writeInt(row);
>> > out.writeInt(col);
>> >
>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>> >
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> >
>> > out.writeDouble(value[i][j]);
>> > }
>> > }
>> > //priting array
>> > for(int vali =0;vali< value.length ;vali ++){
>> > for(int valj = 0;valj <value[0].length;valj++){
>> > System.out.print(value[vali][valj]+ "\t");
>> > }
>> > System.out.println("");
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public void readFields(DataInput in) throws IOException {
>> > int row = in.readInt();
>> > int col = in.readInt();
>> >
>> > double[][] value = new double[row][col];
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> > value[i][j] = in.readDouble();
>> >
>> > }
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public int hashCode() {
>> >
>> > }
>> >
>> > @Override
>> > public boolean equals(Object o) {
>> >
>> > }
>> >
>> >
>> > @Override
>> > public int compareTo(MatrixWritable o) {
>> > // TODO Auto-generated method stub
>> > return 0;
>> > }
>> > @Override
>> > public String toString() {
>> >
>> > return Arrays.toString(value);
>> >
>> > }
>> >
>> >
>> >
>> > }
>> >
>> > I wrote matrix write,readfields and toString.
>> >
>> > 1.But my toString is not returning anything. why is it so?
>> >
>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>> > emiting only custom value- for checking)
>> >
>> > public class MyReducer extends
>> >
>> >
>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>> >
>> > public void reduce(Iterable<MatrixWritable> key,
>> > Iterable<MatrixWritable> values, Context context){
>> > for(MatrixWritable c : values){
>> >
>> > System.out.println("print value "+c.toString());
>> >
>> > }
>> >
>> > }
>> >
>> > but Nothing is printing.when i tried to print value[0].length in
>> toString()
>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i
>> > also needed to print my data asmatrix so i tried
>> >
>> > public String toString() {
>> >
>> > String separator = ", ";
>> > StringBuffer result = new StringBuffer();
>> >
>> > // iterate over the first dimension
>> > for (int i = 0; i < value.length; i++) {
>> > // iterate over the second dimension
>> > for(int j = 0; j < value[i].length; j++){
>> > result.append(value[i][j]);
>> > System.out.print(value[i][j]);
>> > result.append(separator);
>> > }
>> > // remove the last separator
>> > result.setLength(result.length() - separator.length());
>> > // add a line break.
>> > result.append("\n");
>> > }
>> >
>> >
>> > return result.toString();
>> >
>> >
>> > }
>> >
>> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
>> > CompareTo is neccessary right .
>> >
>> > 4.so what should i include in that methods CompareTo,hashcode,equals and
>> > what are these methods intended for.
>> > Any Idea.Pls suggest a solution.
>> >
>> > --
>> > *Thanks & Regards*
>> > *
>> > *
>> > Unmesha Sreeveni U.B*
>> > *
>> > *Junior Developer
>> > *
>> > *Amrita Center For Cyber Security
>> > *
>> > *
>> > Amritapuri.
>> >
>> > www.amrita.edu/cyber/
>> > *
>> >
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Can you share the code?
sent from mobile
On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
> thanks Steve Loughran and Amr Shahin
> Amr Shahin , i refered "
> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
> the same thing only. but my toString is not returning anything.
>
>
>
> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Check this out:
>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>> It shows how to create a customer writable. If you have "hadoop the
>> definitive guide" there is a really good explanation about custom data
>> types.
>>
>> Happy Halloween
>>
>>
>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>> >wrote:
>>
>> > this is my post from stackoverflow
>> > but i am not getting any response.
>> >
>> >
>> > I need to emit a 2D double array as key and value from mapper.There are
>> > questions posted in stackoverflow. But they are not answered.
>> > we have to create a custom datatype.but how?I am new to these custom
>> > datatypes. i dnt have any idea where to start.I am doing some of the
>> matrix
>> > multiplication in a given dataset.and after that i need to emit the
>> value
>> > of
>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so
>> > how to emit these matrices from mapper.And the value should be
>> > corresponding to the key itself.I think for that we need to use
>> > WritableComparable.
>> >
>> >
>> >
>> > public class MatrixWritable implements
>> WritableComparable<MatrixWritable>{
>> >
>> > /**
>> > * @param args
>> > */
>> > private double[][] value;
>> >
>> > public MatrixWritable() {
>> > // TODO Auto-generated constructor stub
>> > set(new double[0][0]);
>> > }
>> >
>> > public MatrixWritable(double[][] value) {
>> > // TODO Auto-generated constructor stub
>> > this.value = value;
>> > }
>> >
>> > public void set(double[][] value) {
>> > this.value = value;
>> > }
>> >
>> > public double[][] getValue() {
>> > return value;
>> > }
>> >
>> > @Override
>> > public void write(DataOutput out) throws IOException {
>> > System.out.println("write");
>> > int row=0;
>> > int col=0;
>> > for(int i=0; i<value.length;i++){
>> > row = value.length;
>> > for(int j=0; j<value[i].length; j++){
>> > col = value[i].length;
>> > }
>> > }
>> > out.writeInt(row);
>> > out.writeInt(col);
>> >
>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>> >
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> >
>> > out.writeDouble(value[i][j]);
>> > }
>> > }
>> > //priting array
>> > for(int vali =0;vali< value.length ;vali ++){
>> > for(int valj = 0;valj <value[0].length;valj++){
>> > System.out.print(value[vali][valj]+ "\t");
>> > }
>> > System.out.println("");
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public void readFields(DataInput in) throws IOException {
>> > int row = in.readInt();
>> > int col = in.readInt();
>> >
>> > double[][] value = new double[row][col];
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> > value[i][j] = in.readDouble();
>> >
>> > }
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public int hashCode() {
>> >
>> > }
>> >
>> > @Override
>> > public boolean equals(Object o) {
>> >
>> > }
>> >
>> >
>> > @Override
>> > public int compareTo(MatrixWritable o) {
>> > // TODO Auto-generated method stub
>> > return 0;
>> > }
>> > @Override
>> > public String toString() {
>> >
>> > return Arrays.toString(value);
>> >
>> > }
>> >
>> >
>> >
>> > }
>> >
>> > I wrote matrix write,readfields and toString.
>> >
>> > 1.But my toString is not returning anything. why is it so?
>> >
>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>> > emiting only custom value- for checking)
>> >
>> > public class MyReducer extends
>> >
>> >
>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>> >
>> > public void reduce(Iterable<MatrixWritable> key,
>> > Iterable<MatrixWritable> values, Context context){
>> > for(MatrixWritable c : values){
>> >
>> > System.out.println("print value "+c.toString());
>> >
>> > }
>> >
>> > }
>> >
>> > but Nothing is printing.when i tried to print value[0].length in
>> toString()
>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i
>> > also needed to print my data asmatrix so i tried
>> >
>> > public String toString() {
>> >
>> > String separator = ", ";
>> > StringBuffer result = new StringBuffer();
>> >
>> > // iterate over the first dimension
>> > for (int i = 0; i < value.length; i++) {
>> > // iterate over the second dimension
>> > for(int j = 0; j < value[i].length; j++){
>> > result.append(value[i][j]);
>> > System.out.print(value[i][j]);
>> > result.append(separator);
>> > }
>> > // remove the last separator
>> > result.setLength(result.length() - separator.length());
>> > // add a line break.
>> > result.append("\n");
>> > }
>> >
>> >
>> > return result.toString();
>> >
>> >
>> > }
>> >
>> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
>> > CompareTo is neccessary right .
>> >
>> > 4.so what should i include in that methods CompareTo,hashcode,equals and
>> > what are these methods intended for.
>> > Any Idea.Pls suggest a solution.
>> >
>> > --
>> > *Thanks & Regards*
>> > *
>> > *
>> > Unmesha Sreeveni U.B*
>> > *
>> > *Junior Developer
>> > *
>> > *Amrita Center For Cyber Security
>> > *
>> > *
>> > Amritapuri.
>> >
>> > www.amrita.edu/cyber/
>> > *
>> >
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Can you share the code?
sent from mobile
On Nov 1, 2013 7:06 AM, "unmesha sreeveni" <un...@gmail.com> wrote:
>
> thanks Steve Loughran and Amr Shahin
> Amr Shahin , i refered "
> http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
> the same thing only. but my toString is not returning anything.
>
>
>
> On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
>
>> Check this out:
>> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
>> It shows how to create a customer writable. If you have "hadoop the
>> definitive guide" there is a really good explanation about custom data
>> types.
>>
>> Happy Halloween
>>
>>
>> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
>> >wrote:
>>
>> > this is my post from stackoverflow
>> > but i am not getting any response.
>> >
>> >
>> > I need to emit a 2D double array as key and value from mapper.There are
>> > questions posted in stackoverflow. But they are not answered.
>> > we have to create a custom datatype.but how?I am new to these custom
>> > datatypes. i dnt have any idea where to start.I am doing some of the
>> matrix
>> > multiplication in a given dataset.and after that i need to emit the
>> value
>> > of
>> > A*Atrns which will be a matrix and Atrans*D which will also be a
>> matrix.so
>> > how to emit these matrices from mapper.And the value should be
>> > corresponding to the key itself.I think for that we need to use
>> > WritableComparable.
>> >
>> >
>> >
>> > public class MatrixWritable implements
>> WritableComparable<MatrixWritable>{
>> >
>> > /**
>> > * @param args
>> > */
>> > private double[][] value;
>> >
>> > public MatrixWritable() {
>> > // TODO Auto-generated constructor stub
>> > set(new double[0][0]);
>> > }
>> >
>> > public MatrixWritable(double[][] value) {
>> > // TODO Auto-generated constructor stub
>> > this.value = value;
>> > }
>> >
>> > public void set(double[][] value) {
>> > this.value = value;
>> > }
>> >
>> > public double[][] getValue() {
>> > return value;
>> > }
>> >
>> > @Override
>> > public void write(DataOutput out) throws IOException {
>> > System.out.println("write");
>> > int row=0;
>> > int col=0;
>> > for(int i=0; i<value.length;i++){
>> > row = value.length;
>> > for(int j=0; j<value[i].length; j++){
>> > col = value[i].length;
>> > }
>> > }
>> > out.writeInt(row);
>> > out.writeInt(col);
>> >
>> > System.out.println("\nTotal no of observations: "+row+":"+col);
>> >
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> >
>> > out.writeDouble(value[i][j]);
>> > }
>> > }
>> > //priting array
>> > for(int vali =0;vali< value.length ;vali ++){
>> > for(int valj = 0;valj <value[0].length;valj++){
>> > System.out.print(value[vali][valj]+ "\t");
>> > }
>> > System.out.println("");
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public void readFields(DataInput in) throws IOException {
>> > int row = in.readInt();
>> > int col = in.readInt();
>> >
>> > double[][] value = new double[row][col];
>> > for(int i=0;i<row ; i++){
>> > for(int j= 0 ; j< col;j++){
>> > value[i][j] = in.readDouble();
>> >
>> > }
>> > }
>> >
>> > }
>> >
>> > @Override
>> > public int hashCode() {
>> >
>> > }
>> >
>> > @Override
>> > public boolean equals(Object o) {
>> >
>> > }
>> >
>> >
>> > @Override
>> > public int compareTo(MatrixWritable o) {
>> > // TODO Auto-generated method stub
>> > return 0;
>> > }
>> > @Override
>> > public String toString() {
>> >
>> > return Arrays.toString(value);
>> >
>> > }
>> >
>> >
>> >
>> > }
>> >
>> > I wrote matrix write,readfields and toString.
>> >
>> > 1.But my toString is not returning anything. why is it so?
>> >
>> > 2.How to print these values with in Reducer ?I tried doing(tried with
>> > emiting only custom value- for checking)
>> >
>> > public class MyReducer extends
>> >
>> >
>> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>> >
>> > public void reduce(Iterable<MatrixWritable> key,
>> > Iterable<MatrixWritable> values, Context context){
>> > for(MatrixWritable c : values){
>> >
>> > System.out.println("print value "+c.toString());
>> >
>> > }
>> >
>> > }
>> >
>> > but Nothing is printing.when i tried to print value[0].length in
>> toString()
>> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and
>> i
>> > also needed to print my data asmatrix so i tried
>> >
>> > public String toString() {
>> >
>> > String separator = ", ";
>> > StringBuffer result = new StringBuffer();
>> >
>> > // iterate over the first dimension
>> > for (int i = 0; i < value.length; i++) {
>> > // iterate over the second dimension
>> > for(int j = 0; j < value[i].length; j++){
>> > result.append(value[i][j]);
>> > System.out.print(value[i][j]);
>> > result.append(separator);
>> > }
>> > // remove the last separator
>> > result.setLength(result.length() - separator.length());
>> > // add a line break.
>> > result.append("\n");
>> > }
>> >
>> >
>> > return result.toString();
>> >
>> >
>> > }
>> >
>> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
>> > CompareTo is neccessary right .
>> >
>> > 4.so what should i include in that methods CompareTo,hashcode,equals and
>> > what are these methods intended for.
>> > Any Idea.Pls suggest a solution.
>> >
>> > --
>> > *Thanks & Regards*
>> > *
>> > *
>> > Unmesha Sreeveni U.B*
>> > *
>> > *Junior Developer
>> > *
>> > *Amrita Center For Cyber Security
>> > *
>> > *
>> > Amritapuri.
>> >
>> > www.amrita.edu/cyber/
>> > *
>> >
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
>
> *Junior Developer*
>
> *Amrita Center For Cyber Security *
>
>
> * Amritapuri.www.amrita.edu/cyber/ <http://www.amrita.edu/cyber/>*
>
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
thanks Steve Loughran and Amr Shahin
Amr Shahin , i refered "
http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
the same thing only. but my toString is not returning anything.
On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
> Check this out:
> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
> It shows how to create a customer writable. If you have "hadoop the
> definitive guide" there is a really good explanation about custom data
> types.
>
> Happy Halloween
>
>
> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
> >wrote:
>
> > this is my post from stackoverflow
> > but i am not getting any response.
> >
> >
> > I need to emit a 2D double array as key and value from mapper.There are
> > questions posted in stackoverflow. But they are not answered.
> > we have to create a custom datatype.but how?I am new to these custom
> > datatypes. i dnt have any idea where to start.I am doing some of the
> matrix
> > multiplication in a given dataset.and after that i need to emit the value
> > of
> > A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so
> > how to emit these matrices from mapper.And the value should be
> > corresponding to the key itself.I think for that we need to use
> > WritableComparable.
> >
> >
> >
> > public class MatrixWritable implements
> WritableComparable<MatrixWritable>{
> >
> > /**
> > * @param args
> > */
> > private double[][] value;
> >
> > public MatrixWritable() {
> > // TODO Auto-generated constructor stub
> > set(new double[0][0]);
> > }
> >
> > public MatrixWritable(double[][] value) {
> > // TODO Auto-generated constructor stub
> > this.value = value;
> > }
> >
> > public void set(double[][] value) {
> > this.value = value;
> > }
> >
> > public double[][] getValue() {
> > return value;
> > }
> >
> > @Override
> > public void write(DataOutput out) throws IOException {
> > System.out.println("write");
> > int row=0;
> > int col=0;
> > for(int i=0; i<value.length;i++){
> > row = value.length;
> > for(int j=0; j<value[i].length; j++){
> > col = value[i].length;
> > }
> > }
> > out.writeInt(row);
> > out.writeInt(col);
> >
> > System.out.println("\nTotal no of observations: "+row+":"+col);
> >
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> >
> > out.writeDouble(value[i][j]);
> > }
> > }
> > //priting array
> > for(int vali =0;vali< value.length ;vali ++){
> > for(int valj = 0;valj <value[0].length;valj++){
> > System.out.print(value[vali][valj]+ "\t");
> > }
> > System.out.println("");
> > }
> >
> > }
> >
> > @Override
> > public void readFields(DataInput in) throws IOException {
> > int row = in.readInt();
> > int col = in.readInt();
> >
> > double[][] value = new double[row][col];
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> > value[i][j] = in.readDouble();
> >
> > }
> > }
> >
> > }
> >
> > @Override
> > public int hashCode() {
> >
> > }
> >
> > @Override
> > public boolean equals(Object o) {
> >
> > }
> >
> >
> > @Override
> > public int compareTo(MatrixWritable o) {
> > // TODO Auto-generated method stub
> > return 0;
> > }
> > @Override
> > public String toString() {
> >
> > return Arrays.toString(value);
> >
> > }
> >
> >
> >
> > }
> >
> > I wrote matrix write,readfields and toString.
> >
> > 1.But my toString is not returning anything. why is it so?
> >
> > 2.How to print these values with in Reducer ?I tried doing(tried with
> > emiting only custom value- for checking)
> >
> > public class MyReducer extends
> >
> >
> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
> >
> > public void reduce(Iterable<MatrixWritable> key,
> > Iterable<MatrixWritable> values, Context context){
> > for(MatrixWritable c : values){
> >
> > System.out.println("print value "+c.toString());
> >
> > }
> >
> > }
> >
> > but Nothing is printing.when i tried to print value[0].length in
> toString()
> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> > also needed to print my data asmatrix so i tried
> >
> > public String toString() {
> >
> > String separator = ", ";
> > StringBuffer result = new StringBuffer();
> >
> > // iterate over the first dimension
> > for (int i = 0; i < value.length; i++) {
> > // iterate over the second dimension
> > for(int j = 0; j < value[i].length; j++){
> > result.append(value[i][j]);
> > System.out.print(value[i][j]);
> > result.append(separator);
> > }
> > // remove the last separator
> > result.setLength(result.length() - separator.length());
> > // add a line break.
> > result.append("\n");
> > }
> >
> >
> > return result.toString();
> >
> >
> > }
> >
> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
> > CompareTo is neccessary right .
> >
> > 4.so what should i include in that methods CompareTo,hashcode,equals and
> > what are these methods intended for.
> > Any Idea.Pls suggest a solution.
> >
> > --
> > *Thanks & Regards*
> > *
> > *
> > Unmesha Sreeveni U.B*
> > *
> > *Junior Developer
> > *
> > *Amrita Center For Cyber Security
> > *
> > *
> > Amritapuri.
> >
> > www.amrita.edu/cyber/
> > *
> >
>
--
*Thanks & Regards*
*
*
Unmesha Sreeveni U.B*
*
*Junior Developer
*
*Amrita Center For Cyber Security
*
*
Amritapuri.
www.amrita.edu/cyber/
*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
thanks Steve Loughran and Amr Shahin
Amr Shahin , i refered "
http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
the same thing only. but my toString is not returning anything.
On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
> Check this out:
> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
> It shows how to create a customer writable. If you have "hadoop the
> definitive guide" there is a really good explanation about custom data
> types.
>
> Happy Halloween
>
>
> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
> >wrote:
>
> > this is my post from stackoverflow
> > but i am not getting any response.
> >
> >
> > I need to emit a 2D double array as key and value from mapper.There are
> > questions posted in stackoverflow. But they are not answered.
> > we have to create a custom datatype.but how?I am new to these custom
> > datatypes. i dnt have any idea where to start.I am doing some of the
> matrix
> > multiplication in a given dataset.and after that i need to emit the value
> > of
> > A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so
> > how to emit these matrices from mapper.And the value should be
> > corresponding to the key itself.I think for that we need to use
> > WritableComparable.
> >
> >
> >
> > public class MatrixWritable implements
> WritableComparable<MatrixWritable>{
> >
> > /**
> > * @param args
> > */
> > private double[][] value;
> >
> > public MatrixWritable() {
> > // TODO Auto-generated constructor stub
> > set(new double[0][0]);
> > }
> >
> > public MatrixWritable(double[][] value) {
> > // TODO Auto-generated constructor stub
> > this.value = value;
> > }
> >
> > public void set(double[][] value) {
> > this.value = value;
> > }
> >
> > public double[][] getValue() {
> > return value;
> > }
> >
> > @Override
> > public void write(DataOutput out) throws IOException {
> > System.out.println("write");
> > int row=0;
> > int col=0;
> > for(int i=0; i<value.length;i++){
> > row = value.length;
> > for(int j=0; j<value[i].length; j++){
> > col = value[i].length;
> > }
> > }
> > out.writeInt(row);
> > out.writeInt(col);
> >
> > System.out.println("\nTotal no of observations: "+row+":"+col);
> >
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> >
> > out.writeDouble(value[i][j]);
> > }
> > }
> > //priting array
> > for(int vali =0;vali< value.length ;vali ++){
> > for(int valj = 0;valj <value[0].length;valj++){
> > System.out.print(value[vali][valj]+ "\t");
> > }
> > System.out.println("");
> > }
> >
> > }
> >
> > @Override
> > public void readFields(DataInput in) throws IOException {
> > int row = in.readInt();
> > int col = in.readInt();
> >
> > double[][] value = new double[row][col];
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> > value[i][j] = in.readDouble();
> >
> > }
> > }
> >
> > }
> >
> > @Override
> > public int hashCode() {
> >
> > }
> >
> > @Override
> > public boolean equals(Object o) {
> >
> > }
> >
> >
> > @Override
> > public int compareTo(MatrixWritable o) {
> > // TODO Auto-generated method stub
> > return 0;
> > }
> > @Override
> > public String toString() {
> >
> > return Arrays.toString(value);
> >
> > }
> >
> >
> >
> > }
> >
> > I wrote matrix write,readfields and toString.
> >
> > 1.But my toString is not returning anything. why is it so?
> >
> > 2.How to print these values with in Reducer ?I tried doing(tried with
> > emiting only custom value- for checking)
> >
> > public class MyReducer extends
> >
> >
> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
> >
> > public void reduce(Iterable<MatrixWritable> key,
> > Iterable<MatrixWritable> values, Context context){
> > for(MatrixWritable c : values){
> >
> > System.out.println("print value "+c.toString());
> >
> > }
> >
> > }
> >
> > but Nothing is printing.when i tried to print value[0].length in
> toString()
> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> > also needed to print my data asmatrix so i tried
> >
> > public String toString() {
> >
> > String separator = ", ";
> > StringBuffer result = new StringBuffer();
> >
> > // iterate over the first dimension
> > for (int i = 0; i < value.length; i++) {
> > // iterate over the second dimension
> > for(int j = 0; j < value[i].length; j++){
> > result.append(value[i][j]);
> > System.out.print(value[i][j]);
> > result.append(separator);
> > }
> > // remove the last separator
> > result.setLength(result.length() - separator.length());
> > // add a line break.
> > result.append("\n");
> > }
> >
> >
> > return result.toString();
> >
> >
> > }
> >
> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
> > CompareTo is neccessary right .
> >
> > 4.so what should i include in that methods CompareTo,hashcode,equals and
> > what are these methods intended for.
> > Any Idea.Pls suggest a solution.
> >
> > --
> > *Thanks & Regards*
> > *
> > *
> > Unmesha Sreeveni U.B*
> > *
> > *Junior Developer
> > *
> > *Amrita Center For Cyber Security
> > *
> > *
> > Amritapuri.
> >
> > www.amrita.edu/cyber/
> > *
> >
>
--
*Thanks & Regards*
*
*
Unmesha Sreeveni U.B*
*
*Junior Developer
*
*Amrita Center For Cyber Security
*
*
Amritapuri.
www.amrita.edu/cyber/
*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
thanks Steve Loughran and Amr Shahin
Amr Shahin , i refered "
http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
the same thing only. but my toString is not returning anything.
On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
> Check this out:
> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
> It shows how to create a customer writable. If you have "hadoop the
> definitive guide" there is a really good explanation about custom data
> types.
>
> Happy Halloween
>
>
> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
> >wrote:
>
> > this is my post from stackoverflow
> > but i am not getting any response.
> >
> >
> > I need to emit a 2D double array as key and value from mapper.There are
> > questions posted in stackoverflow. But they are not answered.
> > we have to create a custom datatype.but how?I am new to these custom
> > datatypes. i dnt have any idea where to start.I am doing some of the
> matrix
> > multiplication in a given dataset.and after that i need to emit the value
> > of
> > A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so
> > how to emit these matrices from mapper.And the value should be
> > corresponding to the key itself.I think for that we need to use
> > WritableComparable.
> >
> >
> >
> > public class MatrixWritable implements
> WritableComparable<MatrixWritable>{
> >
> > /**
> > * @param args
> > */
> > private double[][] value;
> >
> > public MatrixWritable() {
> > // TODO Auto-generated constructor stub
> > set(new double[0][0]);
> > }
> >
> > public MatrixWritable(double[][] value) {
> > // TODO Auto-generated constructor stub
> > this.value = value;
> > }
> >
> > public void set(double[][] value) {
> > this.value = value;
> > }
> >
> > public double[][] getValue() {
> > return value;
> > }
> >
> > @Override
> > public void write(DataOutput out) throws IOException {
> > System.out.println("write");
> > int row=0;
> > int col=0;
> > for(int i=0; i<value.length;i++){
> > row = value.length;
> > for(int j=0; j<value[i].length; j++){
> > col = value[i].length;
> > }
> > }
> > out.writeInt(row);
> > out.writeInt(col);
> >
> > System.out.println("\nTotal no of observations: "+row+":"+col);
> >
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> >
> > out.writeDouble(value[i][j]);
> > }
> > }
> > //priting array
> > for(int vali =0;vali< value.length ;vali ++){
> > for(int valj = 0;valj <value[0].length;valj++){
> > System.out.print(value[vali][valj]+ "\t");
> > }
> > System.out.println("");
> > }
> >
> > }
> >
> > @Override
> > public void readFields(DataInput in) throws IOException {
> > int row = in.readInt();
> > int col = in.readInt();
> >
> > double[][] value = new double[row][col];
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> > value[i][j] = in.readDouble();
> >
> > }
> > }
> >
> > }
> >
> > @Override
> > public int hashCode() {
> >
> > }
> >
> > @Override
> > public boolean equals(Object o) {
> >
> > }
> >
> >
> > @Override
> > public int compareTo(MatrixWritable o) {
> > // TODO Auto-generated method stub
> > return 0;
> > }
> > @Override
> > public String toString() {
> >
> > return Arrays.toString(value);
> >
> > }
> >
> >
> >
> > }
> >
> > I wrote matrix write,readfields and toString.
> >
> > 1.But my toString is not returning anything. why is it so?
> >
> > 2.How to print these values with in Reducer ?I tried doing(tried with
> > emiting only custom value- for checking)
> >
> > public class MyReducer extends
> >
> >
> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
> >
> > public void reduce(Iterable<MatrixWritable> key,
> > Iterable<MatrixWritable> values, Context context){
> > for(MatrixWritable c : values){
> >
> > System.out.println("print value "+c.toString());
> >
> > }
> >
> > }
> >
> > but Nothing is printing.when i tried to print value[0].length in
> toString()
> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> > also needed to print my data asmatrix so i tried
> >
> > public String toString() {
> >
> > String separator = ", ";
> > StringBuffer result = new StringBuffer();
> >
> > // iterate over the first dimension
> > for (int i = 0; i < value.length; i++) {
> > // iterate over the second dimension
> > for(int j = 0; j < value[i].length; j++){
> > result.append(value[i][j]);
> > System.out.print(value[i][j]);
> > result.append(separator);
> > }
> > // remove the last separator
> > result.setLength(result.length() - separator.length());
> > // add a line break.
> > result.append("\n");
> > }
> >
> >
> > return result.toString();
> >
> >
> > }
> >
> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
> > CompareTo is neccessary right .
> >
> > 4.so what should i include in that methods CompareTo,hashcode,equals and
> > what are these methods intended for.
> > Any Idea.Pls suggest a solution.
> >
> > --
> > *Thanks & Regards*
> > *
> > *
> > Unmesha Sreeveni U.B*
> > *
> > *Junior Developer
> > *
> > *Amrita Center For Cyber Security
> > *
> > *
> > Amritapuri.
> >
> > www.amrita.edu/cyber/
> > *
> >
>
--
*Thanks & Regards*
*
*
Unmesha Sreeveni U.B*
*
*Junior Developer
*
*Amrita Center For Cyber Security
*
*
Amritapuri.
www.amrita.edu/cyber/
*
Re: Implementing a custom hadoop key and value - need Help
Posted by unmesha sreeveni <un...@gmail.com>.
thanks Steve Loughran and Amr Shahin
Amr Shahin , i refered "
http://my.safaribooksonline.com/book/databases/hadoop/9780596521974/serialization/id3548156"
the same thing only. but my toString is not returning anything.
On Thu, Oct 31, 2013 at 5:57 PM, Amr Shahin <am...@gmail.com> wrote:
> Check this out:
> http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
> It shows how to create a customer writable. If you have "hadoop the
> definitive guide" there is a really good explanation about custom data
> types.
>
> Happy Halloween
>
>
> On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <unmeshabiju@gmail.com
> >wrote:
>
> > this is my post from stackoverflow
> > but i am not getting any response.
> >
> >
> > I need to emit a 2D double array as key and value from mapper.There are
> > questions posted in stackoverflow. But they are not answered.
> > we have to create a custom datatype.but how?I am new to these custom
> > datatypes. i dnt have any idea where to start.I am doing some of the
> matrix
> > multiplication in a given dataset.and after that i need to emit the value
> > of
> > A*Atrns which will be a matrix and Atrans*D which will also be a
> matrix.so
> > how to emit these matrices from mapper.And the value should be
> > corresponding to the key itself.I think for that we need to use
> > WritableComparable.
> >
> >
> >
> > public class MatrixWritable implements
> WritableComparable<MatrixWritable>{
> >
> > /**
> > * @param args
> > */
> > private double[][] value;
> >
> > public MatrixWritable() {
> > // TODO Auto-generated constructor stub
> > set(new double[0][0]);
> > }
> >
> > public MatrixWritable(double[][] value) {
> > // TODO Auto-generated constructor stub
> > this.value = value;
> > }
> >
> > public void set(double[][] value) {
> > this.value = value;
> > }
> >
> > public double[][] getValue() {
> > return value;
> > }
> >
> > @Override
> > public void write(DataOutput out) throws IOException {
> > System.out.println("write");
> > int row=0;
> > int col=0;
> > for(int i=0; i<value.length;i++){
> > row = value.length;
> > for(int j=0; j<value[i].length; j++){
> > col = value[i].length;
> > }
> > }
> > out.writeInt(row);
> > out.writeInt(col);
> >
> > System.out.println("\nTotal no of observations: "+row+":"+col);
> >
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> >
> > out.writeDouble(value[i][j]);
> > }
> > }
> > //priting array
> > for(int vali =0;vali< value.length ;vali ++){
> > for(int valj = 0;valj <value[0].length;valj++){
> > System.out.print(value[vali][valj]+ "\t");
> > }
> > System.out.println("");
> > }
> >
> > }
> >
> > @Override
> > public void readFields(DataInput in) throws IOException {
> > int row = in.readInt();
> > int col = in.readInt();
> >
> > double[][] value = new double[row][col];
> > for(int i=0;i<row ; i++){
> > for(int j= 0 ; j< col;j++){
> > value[i][j] = in.readDouble();
> >
> > }
> > }
> >
> > }
> >
> > @Override
> > public int hashCode() {
> >
> > }
> >
> > @Override
> > public boolean equals(Object o) {
> >
> > }
> >
> >
> > @Override
> > public int compareTo(MatrixWritable o) {
> > // TODO Auto-generated method stub
> > return 0;
> > }
> > @Override
> > public String toString() {
> >
> > return Arrays.toString(value);
> >
> > }
> >
> >
> >
> > }
> >
> > I wrote matrix write,readfields and toString.
> >
> > 1.But my toString is not returning anything. why is it so?
> >
> > 2.How to print these values with in Reducer ?I tried doing(tried with
> > emiting only custom value- for checking)
> >
> > public class MyReducer extends
> >
> >
> > Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
> >
> > public void reduce(Iterable<MatrixWritable> key,
> > Iterable<MatrixWritable> values, Context context){
> > for(MatrixWritable c : values){
> >
> > System.out.println("print value "+c.toString());
> >
> > }
> >
> > }
> >
> > but Nothing is printing.when i tried to print value[0].length in
> toString()
> > method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> > also needed to print my data asmatrix so i tried
> >
> > public String toString() {
> >
> > String separator = ", ";
> > StringBuffer result = new StringBuffer();
> >
> > // iterate over the first dimension
> > for (int i = 0; i < value.length; i++) {
> > // iterate over the second dimension
> > for(int j = 0; j < value[i].length; j++){
> > result.append(value[i][j]);
> > System.out.print(value[i][j]);
> > result.append(separator);
> > }
> > // remove the last separator
> > result.setLength(result.length() - separator.length());
> > // add a line break.
> > result.append("\n");
> > }
> >
> >
> > return result.toString();
> >
> >
> > }
> >
> > Again my output is empty. 3.Inorder to emit a key too as custom datatype
> > CompareTo is neccessary right .
> >
> > 4.so what should i include in that methods CompareTo,hashcode,equals and
> > what are these methods intended for.
> > Any Idea.Pls suggest a solution.
> >
> > --
> > *Thanks & Regards*
> > *
> > *
> > Unmesha Sreeveni U.B*
> > *
> > *Junior Developer
> > *
> > *Amrita Center For Cyber Security
> > *
> > *
> > Amritapuri.
> >
> > www.amrita.edu/cyber/
> > *
> >
>
--
*Thanks & Regards*
*
*
Unmesha Sreeveni U.B*
*
*Junior Developer
*
*Amrita Center For Cyber Security
*
*
Amritapuri.
www.amrita.edu/cyber/
*
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
It shows how to create a customer writable. If you have "hadoop the
definitive guide" there is a really good explanation about custom data
types.
Happy Halloween
On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <un...@gmail.com>wrote:
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value
> of
> A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with
> emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
> Again my output is empty. 3.Inorder to emit a key too as custom datatype
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals and
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.edu/cyber/
> *
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
It shows how to create a customer writable. If you have "hadoop the
definitive guide" there is a really good explanation about custom data
types.
Happy Halloween
On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <un...@gmail.com>wrote:
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value
> of
> A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with
> emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
> Again my output is empty. 3.Inorder to emit a key too as custom datatype
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals and
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.edu/cyber/
> *
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
It shows how to create a customer writable. If you have "hadoop the
definitive guide" there is a really good explanation about custom data
types.
Happy Halloween
On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <un...@gmail.com>wrote:
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value
> of
> A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with
> emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
> Again my output is empty. 3.Inorder to emit a key too as custom datatype
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals and
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.edu/cyber/
> *
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
It shows how to create a customer writable. If you have "hadoop the
definitive guide" there is a really good explanation about custom data
types.
Happy Halloween
On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <un...@gmail.com>wrote:
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value
> of
> A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with
> emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
> Again my output is empty. 3.Inorder to emit a key too as custom datatype
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals and
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.edu/cyber/
> *
>
Re: Implementing a custom hadoop key and value - need Help
Posted by Steve Loughran <st...@apache.org>.
On 31 October 2013 11:03, unmesha sreeveni <un...@gmail.com> wrote:
> this is my post from stackoverflow
> but i am not getting any response.
>
Well, its unlikely you will get any response from hadoop general or common
dev as they are not for end user questions.
please look at the guide to which mailing list to use, post to the
appropriate one.
http://hadoop.apache.org/mailing_lists.html
Also: silence means nobody knows the answer and/or has the time to debug
your code, reposting isn't going to help. Sorry
steve
Re: Implementing a custom hadoop key and value - need Help
Posted by Amr Shahin <am...@gmail.com>.
Check this out:
http://developer.yahoo.com/hadoop/tutorial/module5.html#writable-notes
It shows how to create a customer writable. If you have "hadoop the
definitive guide" there is a really good explanation about custom data
types.
Happy Halloween
On Thu, Oct 31, 2013 at 3:03 PM, unmesha sreeveni <un...@gmail.com>wrote:
> this is my post from stackoverflow
> but i am not getting any response.
>
>
> I need to emit a 2D double array as key and value from mapper.There are
> questions posted in stackoverflow. But they are not answered.
> we have to create a custom datatype.but how?I am new to these custom
> datatypes. i dnt have any idea where to start.I am doing some of the matrix
> multiplication in a given dataset.and after that i need to emit the value
> of
> A*Atrns which will be a matrix and Atrans*D which will also be a matrix.so
> how to emit these matrices from mapper.And the value should be
> corresponding to the key itself.I think for that we need to use
> WritableComparable.
>
>
>
> public class MatrixWritable implements WritableComparable<MatrixWritable>{
>
> /**
> * @param args
> */
> private double[][] value;
>
> public MatrixWritable() {
> // TODO Auto-generated constructor stub
> set(new double[0][0]);
> }
>
> public MatrixWritable(double[][] value) {
> // TODO Auto-generated constructor stub
> this.value = value;
> }
>
> public void set(double[][] value) {
> this.value = value;
> }
>
> public double[][] getValue() {
> return value;
> }
>
> @Override
> public void write(DataOutput out) throws IOException {
> System.out.println("write");
> int row=0;
> int col=0;
> for(int i=0; i<value.length;i++){
> row = value.length;
> for(int j=0; j<value[i].length; j++){
> col = value[i].length;
> }
> }
> out.writeInt(row);
> out.writeInt(col);
>
> System.out.println("\nTotal no of observations: "+row+":"+col);
>
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
>
> out.writeDouble(value[i][j]);
> }
> }
> //priting array
> for(int vali =0;vali< value.length ;vali ++){
> for(int valj = 0;valj <value[0].length;valj++){
> System.out.print(value[vali][valj]+ "\t");
> }
> System.out.println("");
> }
>
> }
>
> @Override
> public void readFields(DataInput in) throws IOException {
> int row = in.readInt();
> int col = in.readInt();
>
> double[][] value = new double[row][col];
> for(int i=0;i<row ; i++){
> for(int j= 0 ; j< col;j++){
> value[i][j] = in.readDouble();
>
> }
> }
>
> }
>
> @Override
> public int hashCode() {
>
> }
>
> @Override
> public boolean equals(Object o) {
>
> }
>
>
> @Override
> public int compareTo(MatrixWritable o) {
> // TODO Auto-generated method stub
> return 0;
> }
> @Override
> public String toString() {
>
> return Arrays.toString(value);
>
> }
>
>
>
> }
>
> I wrote matrix write,readfields and toString.
>
> 1.But my toString is not returning anything. why is it so?
>
> 2.How to print these values with in Reducer ?I tried doing(tried with
> emiting only custom value- for checking)
>
> public class MyReducer extends
>
>
> Reducer<MatrixWritable, MatrixWritable, IntWritable, Text> {
>
> public void reduce(Iterable<MatrixWritable> key,
> Iterable<MatrixWritable> values, Context context){
> for(MatrixWritable c : values){
>
> System.out.println("print value "+c.toString());
>
> }
>
> }
>
> but Nothing is printing.when i tried to print value[0].length in toString()
> method it showsArrayIndexOutOfBoundExcep.Am i doing any thing wrong.and i
> also needed to print my data asmatrix so i tried
>
> public String toString() {
>
> String separator = ", ";
> StringBuffer result = new StringBuffer();
>
> // iterate over the first dimension
> for (int i = 0; i < value.length; i++) {
> // iterate over the second dimension
> for(int j = 0; j < value[i].length; j++){
> result.append(value[i][j]);
> System.out.print(value[i][j]);
> result.append(separator);
> }
> // remove the last separator
> result.setLength(result.length() - separator.length());
> // add a line break.
> result.append("\n");
> }
>
>
> return result.toString();
>
>
> }
>
> Again my output is empty. 3.Inorder to emit a key too as custom datatype
> CompareTo is neccessary right .
>
> 4.so what should i include in that methods CompareTo,hashcode,equals and
> what are these methods intended for.
> Any Idea.Pls suggest a solution.
>
> --
> *Thanks & Regards*
> *
> *
> Unmesha Sreeveni U.B*
> *
> *Junior Developer
> *
> *Amrita Center For Cyber Security
> *
> *
> Amritapuri.
>
> www.amrita.edu/cyber/
> *
>