You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by unmesha sreeveni <un...@gmail.com> on 2015/02/21 08:55:19 UTC
Get method in Writable
Am I able to get the values from writable of a previous job.
ie I have 2 MR jobs
*MR 1:*
I need to pass 3 element as values from reducer and the key is
NullWritable. So I created a custom writable class to achieve this.
* public class TreeInfoWritable implements Writable{*
* DoubleWritable entropy;*
* IntWritable sum;*
* IntWritable clsCount;*
* ..*
*}*
*MR 2:*
I need to access MR 1 result in MR2 mapper setup function. And I accessed
it as distributed cache (small file).
Is there a way to get those values using get method.
*while ((setupData = bf.readLine()) != null) {*
* System.out.println("Setup Line "+setupData);*
* TreeInfoWritable info = //something i can pass to TreeInfoWritable and
get values*
* DoubleWritable entropy = info.getEntropy();*
* System.out.println("entropy: "+entropy);*
*}*
Tried to convert writable to gson format.
*MR 1*
*Gson gson = new Gson();*
*String emitVal = gson.toJson(valEmit);*
*context.write(out, new Text(emitVal));*
But parsing canot be done in *MR2*.
*TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
*Error: Type mismatch: cannot convert from String to TreeInfoWritable*
Once it is changed to string we cannot get values.
Am I able to get a workaround for the same. or to use just POJO classes
instaed of Writable. I'm afraid if that becomes slower as we are depending
on Java instaed of hadoop 's serializable classes
Re: Get method in Writable
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks Drake.
That was the point.It was my mistake.
On Mon, Feb 23, 2015 at 6:34 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi, unmesha.
>
> I think this is a gson problem. you mentioned like this:
>
> >> But parsing canot be done in *MR2*.
> *>> TreeInfoWritable info = gson.toJson(setupData,
> TreeInfoWritable.class);*
>
> I think just use gson.fromJson, not toJson(setupData is already json
> string, i think).
>
> Is this right ?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
> wrote:
>
>> Am I able to get the values from writable of a previous job.
>> ie I have 2 MR jobs
>> *MR 1:*
>> I need to pass 3 element as values from reducer and the key is
>> NullWritable. So I created a custom writable class to achieve this.
>> * public class TreeInfoWritable implements Writable{*
>>
>> * DoubleWritable entropy;*
>> * IntWritable sum;*
>> * IntWritable clsCount;*
>> * ..*
>> *}*
>> *MR 2:*
>> I need to access MR 1 result in MR2 mapper setup function. And I
>> accessed it as distributed cache (small file).
>> Is there a way to get those values using get method.
>> *while ((setupData = bf.readLine()) != null) {*
>> * System.out.println("Setup Line "+setupData);*
>> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
>> get values*
>> * DoubleWritable entropy = info.getEntropy();*
>> * System.out.println("entropy: "+entropy);*
>> *}*
>>
>> Tried to convert writable to gson format.
>> *MR 1*
>> *Gson gson = new Gson();*
>> *String emitVal = gson.toJson(valEmit);*
>> *context.write(out, new Text(emitVal));*
>>
>> But parsing canot be done in *MR2*.
>> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>>
>> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
>> Once it is changed to string we cannot get values.
>>
>> Am I able to get a workaround for the same. or to use just POJO classes
>> instaed of Writable. I'm afraid if that becomes slower as we are depending
>> on Java instaed of hadoop 's serializable classes
>>
>>
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Get method in Writable
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks Drake.
That was the point.It was my mistake.
On Mon, Feb 23, 2015 at 6:34 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi, unmesha.
>
> I think this is a gson problem. you mentioned like this:
>
> >> But parsing canot be done in *MR2*.
> *>> TreeInfoWritable info = gson.toJson(setupData,
> TreeInfoWritable.class);*
>
> I think just use gson.fromJson, not toJson(setupData is already json
> string, i think).
>
> Is this right ?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
> wrote:
>
>> Am I able to get the values from writable of a previous job.
>> ie I have 2 MR jobs
>> *MR 1:*
>> I need to pass 3 element as values from reducer and the key is
>> NullWritable. So I created a custom writable class to achieve this.
>> * public class TreeInfoWritable implements Writable{*
>>
>> * DoubleWritable entropy;*
>> * IntWritable sum;*
>> * IntWritable clsCount;*
>> * ..*
>> *}*
>> *MR 2:*
>> I need to access MR 1 result in MR2 mapper setup function. And I
>> accessed it as distributed cache (small file).
>> Is there a way to get those values using get method.
>> *while ((setupData = bf.readLine()) != null) {*
>> * System.out.println("Setup Line "+setupData);*
>> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
>> get values*
>> * DoubleWritable entropy = info.getEntropy();*
>> * System.out.println("entropy: "+entropy);*
>> *}*
>>
>> Tried to convert writable to gson format.
>> *MR 1*
>> *Gson gson = new Gson();*
>> *String emitVal = gson.toJson(valEmit);*
>> *context.write(out, new Text(emitVal));*
>>
>> But parsing canot be done in *MR2*.
>> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>>
>> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
>> Once it is changed to string we cannot get values.
>>
>> Am I able to get a workaround for the same. or to use just POJO classes
>> instaed of Writable. I'm afraid if that becomes slower as we are depending
>> on Java instaed of hadoop 's serializable classes
>>
>>
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Get method in Writable
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks Drake.
That was the point.It was my mistake.
On Mon, Feb 23, 2015 at 6:34 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi, unmesha.
>
> I think this is a gson problem. you mentioned like this:
>
> >> But parsing canot be done in *MR2*.
> *>> TreeInfoWritable info = gson.toJson(setupData,
> TreeInfoWritable.class);*
>
> I think just use gson.fromJson, not toJson(setupData is already json
> string, i think).
>
> Is this right ?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
> wrote:
>
>> Am I able to get the values from writable of a previous job.
>> ie I have 2 MR jobs
>> *MR 1:*
>> I need to pass 3 element as values from reducer and the key is
>> NullWritable. So I created a custom writable class to achieve this.
>> * public class TreeInfoWritable implements Writable{*
>>
>> * DoubleWritable entropy;*
>> * IntWritable sum;*
>> * IntWritable clsCount;*
>> * ..*
>> *}*
>> *MR 2:*
>> I need to access MR 1 result in MR2 mapper setup function. And I
>> accessed it as distributed cache (small file).
>> Is there a way to get those values using get method.
>> *while ((setupData = bf.readLine()) != null) {*
>> * System.out.println("Setup Line "+setupData);*
>> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
>> get values*
>> * DoubleWritable entropy = info.getEntropy();*
>> * System.out.println("entropy: "+entropy);*
>> *}*
>>
>> Tried to convert writable to gson format.
>> *MR 1*
>> *Gson gson = new Gson();*
>> *String emitVal = gson.toJson(valEmit);*
>> *context.write(out, new Text(emitVal));*
>>
>> But parsing canot be done in *MR2*.
>> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>>
>> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
>> Once it is changed to string we cannot get values.
>>
>> Am I able to get a workaround for the same. or to use just POJO classes
>> instaed of Writable. I'm afraid if that becomes slower as we are depending
>> on Java instaed of hadoop 's serializable classes
>>
>>
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Get method in Writable
Posted by unmesha sreeveni <un...@gmail.com>.
Thanks Drake.
That was the point.It was my mistake.
On Mon, Feb 23, 2015 at 6:34 AM, Drake민영근 <dr...@nexr.com> wrote:
> Hi, unmesha.
>
> I think this is a gson problem. you mentioned like this:
>
> >> But parsing canot be done in *MR2*.
> *>> TreeInfoWritable info = gson.toJson(setupData,
> TreeInfoWritable.class);*
>
> I think just use gson.fromJson, not toJson(setupData is already json
> string, i think).
>
> Is this right ?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
> wrote:
>
>> Am I able to get the values from writable of a previous job.
>> ie I have 2 MR jobs
>> *MR 1:*
>> I need to pass 3 element as values from reducer and the key is
>> NullWritable. So I created a custom writable class to achieve this.
>> * public class TreeInfoWritable implements Writable{*
>>
>> * DoubleWritable entropy;*
>> * IntWritable sum;*
>> * IntWritable clsCount;*
>> * ..*
>> *}*
>> *MR 2:*
>> I need to access MR 1 result in MR2 mapper setup function. And I
>> accessed it as distributed cache (small file).
>> Is there a way to get those values using get method.
>> *while ((setupData = bf.readLine()) != null) {*
>> * System.out.println("Setup Line "+setupData);*
>> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
>> get values*
>> * DoubleWritable entropy = info.getEntropy();*
>> * System.out.println("entropy: "+entropy);*
>> *}*
>>
>> Tried to convert writable to gson format.
>> *MR 1*
>> *Gson gson = new Gson();*
>> *String emitVal = gson.toJson(valEmit);*
>> *context.write(out, new Text(emitVal));*
>>
>> But parsing canot be done in *MR2*.
>> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>>
>> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
>> Once it is changed to string we cannot get values.
>>
>> Am I able to get a workaround for the same. or to use just POJO classes
>> instaed of Writable. I'm afraid if that becomes slower as we are depending
>> on Java instaed of hadoop 's serializable classes
>>
>>
>>
>>
>>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Centre for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Get method in Writable
Posted by Drake민영근 <dr...@nexr.com>.
Hi, unmesha.
I think this is a gson problem. you mentioned like this:
>> But parsing canot be done in *MR2*.
*>> TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
I think just use gson.fromJson, not toJson(setupData is already json
string, i think).
Is this right ?
Drake 민영근 Ph.D
kt NexR
On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Am I able to get the values from writable of a previous job.
> ie I have 2 MR jobs
> *MR 1:*
> I need to pass 3 element as values from reducer and the key is
> NullWritable. So I created a custom writable class to achieve this.
> * public class TreeInfoWritable implements Writable{*
>
> * DoubleWritable entropy;*
> * IntWritable sum;*
> * IntWritable clsCount;*
> * ..*
> *}*
> *MR 2:*
> I need to access MR 1 result in MR2 mapper setup function. And I accessed
> it as distributed cache (small file).
> Is there a way to get those values using get method.
> *while ((setupData = bf.readLine()) != null) {*
> * System.out.println("Setup Line "+setupData);*
> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
> get values*
> * DoubleWritable entropy = info.getEntropy();*
> * System.out.println("entropy: "+entropy);*
> *}*
>
> Tried to convert writable to gson format.
> *MR 1*
> *Gson gson = new Gson();*
> *String emitVal = gson.toJson(valEmit);*
> *context.write(out, new Text(emitVal));*
>
> But parsing canot be done in *MR2*.
> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>
> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
> Once it is changed to string we cannot get values.
>
> Am I able to get a workaround for the same. or to use just POJO classes
> instaed of Writable. I'm afraid if that becomes slower as we are depending
> on Java instaed of hadoop 's serializable classes
>
>
>
>
>
Re: Get method in Writable
Posted by Drake민영근 <dr...@nexr.com>.
Hi, unmesha.
I think this is a gson problem. you mentioned like this:
>> But parsing canot be done in *MR2*.
*>> TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
I think just use gson.fromJson, not toJson(setupData is already json
string, i think).
Is this right ?
Drake 민영근 Ph.D
kt NexR
On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Am I able to get the values from writable of a previous job.
> ie I have 2 MR jobs
> *MR 1:*
> I need to pass 3 element as values from reducer and the key is
> NullWritable. So I created a custom writable class to achieve this.
> * public class TreeInfoWritable implements Writable{*
>
> * DoubleWritable entropy;*
> * IntWritable sum;*
> * IntWritable clsCount;*
> * ..*
> *}*
> *MR 2:*
> I need to access MR 1 result in MR2 mapper setup function. And I accessed
> it as distributed cache (small file).
> Is there a way to get those values using get method.
> *while ((setupData = bf.readLine()) != null) {*
> * System.out.println("Setup Line "+setupData);*
> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
> get values*
> * DoubleWritable entropy = info.getEntropy();*
> * System.out.println("entropy: "+entropy);*
> *}*
>
> Tried to convert writable to gson format.
> *MR 1*
> *Gson gson = new Gson();*
> *String emitVal = gson.toJson(valEmit);*
> *context.write(out, new Text(emitVal));*
>
> But parsing canot be done in *MR2*.
> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>
> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
> Once it is changed to string we cannot get values.
>
> Am I able to get a workaround for the same. or to use just POJO classes
> instaed of Writable. I'm afraid if that becomes slower as we are depending
> on Java instaed of hadoop 's serializable classes
>
>
>
>
>
Re: Get method in Writable
Posted by Drake민영근 <dr...@nexr.com>.
Hi, unmesha.
I think this is a gson problem. you mentioned like this:
>> But parsing canot be done in *MR2*.
*>> TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
I think just use gson.fromJson, not toJson(setupData is already json
string, i think).
Is this right ?
Drake 민영근 Ph.D
kt NexR
On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Am I able to get the values from writable of a previous job.
> ie I have 2 MR jobs
> *MR 1:*
> I need to pass 3 element as values from reducer and the key is
> NullWritable. So I created a custom writable class to achieve this.
> * public class TreeInfoWritable implements Writable{*
>
> * DoubleWritable entropy;*
> * IntWritable sum;*
> * IntWritable clsCount;*
> * ..*
> *}*
> *MR 2:*
> I need to access MR 1 result in MR2 mapper setup function. And I accessed
> it as distributed cache (small file).
> Is there a way to get those values using get method.
> *while ((setupData = bf.readLine()) != null) {*
> * System.out.println("Setup Line "+setupData);*
> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
> get values*
> * DoubleWritable entropy = info.getEntropy();*
> * System.out.println("entropy: "+entropy);*
> *}*
>
> Tried to convert writable to gson format.
> *MR 1*
> *Gson gson = new Gson();*
> *String emitVal = gson.toJson(valEmit);*
> *context.write(out, new Text(emitVal));*
>
> But parsing canot be done in *MR2*.
> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>
> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
> Once it is changed to string we cannot get values.
>
> Am I able to get a workaround for the same. or to use just POJO classes
> instaed of Writable. I'm afraid if that becomes slower as we are depending
> on Java instaed of hadoop 's serializable classes
>
>
>
>
>
Re: Get method in Writable
Posted by Drake민영근 <dr...@nexr.com>.
Hi, unmesha.
I think this is a gson problem. you mentioned like this:
>> But parsing canot be done in *MR2*.
*>> TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
I think just use gson.fromJson, not toJson(setupData is already json
string, i think).
Is this right ?
Drake 민영근 Ph.D
kt NexR
On Sat, Feb 21, 2015 at 4:55 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Am I able to get the values from writable of a previous job.
> ie I have 2 MR jobs
> *MR 1:*
> I need to pass 3 element as values from reducer and the key is
> NullWritable. So I created a custom writable class to achieve this.
> * public class TreeInfoWritable implements Writable{*
>
> * DoubleWritable entropy;*
> * IntWritable sum;*
> * IntWritable clsCount;*
> * ..*
> *}*
> *MR 2:*
> I need to access MR 1 result in MR2 mapper setup function. And I accessed
> it as distributed cache (small file).
> Is there a way to get those values using get method.
> *while ((setupData = bf.readLine()) != null) {*
> * System.out.println("Setup Line "+setupData);*
> * TreeInfoWritable info = //something i can pass to TreeInfoWritable and
> get values*
> * DoubleWritable entropy = info.getEntropy();*
> * System.out.println("entropy: "+entropy);*
> *}*
>
> Tried to convert writable to gson format.
> *MR 1*
> *Gson gson = new Gson();*
> *String emitVal = gson.toJson(valEmit);*
> *context.write(out, new Text(emitVal));*
>
> But parsing canot be done in *MR2*.
> *TreeInfoWritable info = gson.toJson(setupData, TreeInfoWritable.class);*
>
> *Error: Type mismatch: cannot convert from String to TreeInfoWritable*
> Once it is changed to string we cannot get values.
>
> Am I able to get a workaround for the same. or to use just POJO classes
> instaed of Writable. I'm afraid if that becomes slower as we are depending
> on Java instaed of hadoop 's serializable classes
>
>
>
>
>