You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Vikas Jadhav <vi...@gmail.com> on 2013/04/20 17:43:21 UTC
Writing intermediate key,value pairs to file and read it again
Hello,
Can anyone help me in following issue
Writing intermediate key,value pairs to file and read it again
let us say i have to write each intermediate pair received @reducer to a
file and again read that as key value pair again and use it for processing
I found IFile.java file which has reader and writer but i am not able
understand how to use it for example. I dont understand Counter value as
last parameter "spilledRecordsCounter"
Thanks.
--
*
*
*
Regards,*
* Vikas *
Re: Writing intermediate key,value pairs to file and read it again
Posted by Vikas Jadhav <vi...@gmail.com>.
ya i looked @ chain reducer but it will not help in my case
Now i am able to write each key value pair received @ reducer(using
IFile.Writer )
but when i try to read it using IFile.Reader it reads all data not single
key value pair
Example:
*IFile.Reader reader = new* IFile.Reader<KEYIN,
VALUEIN>(context.getConfiguration(), localFs, file, *null*, myReadCounter);
DataInputBuffer keyRead = new DataInputBuffer();
DataInputBuffer valueRead = new DataInputBuffer();
reader.next(keyRead, valueRead);
when i say keyRead.getData()
it print all key value pair in that buffer i want single key value pair.
On Sun, Apr 21, 2013 at 9:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> you would look at chain reducer java doc, which meet your requirement.
>
> --Send from my Sony mobile.
> On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
>
>> Hello,
>> Can anyone help me in following issue
>> Writing intermediate key,value pairs to file and read it again
>>
>> let us say i have to write each intermediate pair received @reducer to a
>> file and again read that as key value pair again and use it for processing
>>
>> I found IFile.java file which has reader and writer but i am not able
>> understand how to use it for example. I dont understand Counter value as
>> last parameter "spilledRecordsCounter"
>>
>>
>> Thanks.
>>
>>
>> --
>> *
>> *
>> *
>>
>> Regards,*
>> * Vikas *
>>
>
--
*
*
*
Regards,*
* Vikas *
Re: Writing intermediate key,value pairs to file and read it again
Posted by Vikas Jadhav <vi...@gmail.com>.
ya i looked @ chain reducer but it will not help in my case
Now i am able to write each key value pair received @ reducer(using
IFile.Writer )
but when i try to read it using IFile.Reader it reads all data not single
key value pair
Example:
*IFile.Reader reader = new* IFile.Reader<KEYIN,
VALUEIN>(context.getConfiguration(), localFs, file, *null*, myReadCounter);
DataInputBuffer keyRead = new DataInputBuffer();
DataInputBuffer valueRead = new DataInputBuffer();
reader.next(keyRead, valueRead);
when i say keyRead.getData()
it print all key value pair in that buffer i want single key value pair.
On Sun, Apr 21, 2013 at 9:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> you would look at chain reducer java doc, which meet your requirement.
>
> --Send from my Sony mobile.
> On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
>
>> Hello,
>> Can anyone help me in following issue
>> Writing intermediate key,value pairs to file and read it again
>>
>> let us say i have to write each intermediate pair received @reducer to a
>> file and again read that as key value pair again and use it for processing
>>
>> I found IFile.java file which has reader and writer but i am not able
>> understand how to use it for example. I dont understand Counter value as
>> last parameter "spilledRecordsCounter"
>>
>>
>> Thanks.
>>
>>
>> --
>> *
>> *
>> *
>>
>> Regards,*
>> * Vikas *
>>
>
--
*
*
*
Regards,*
* Vikas *
Re: Writing intermediate key,value pairs to file and read it again
Posted by Vikas Jadhav <vi...@gmail.com>.
ya i looked @ chain reducer but it will not help in my case
Now i am able to write each key value pair received @ reducer(using
IFile.Writer )
but when i try to read it using IFile.Reader it reads all data not single
key value pair
Example:
*IFile.Reader reader = new* IFile.Reader<KEYIN,
VALUEIN>(context.getConfiguration(), localFs, file, *null*, myReadCounter);
DataInputBuffer keyRead = new DataInputBuffer();
DataInputBuffer valueRead = new DataInputBuffer();
reader.next(keyRead, valueRead);
when i say keyRead.getData()
it print all key value pair in that buffer i want single key value pair.
On Sun, Apr 21, 2013 at 9:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> you would look at chain reducer java doc, which meet your requirement.
>
> --Send from my Sony mobile.
> On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
>
>> Hello,
>> Can anyone help me in following issue
>> Writing intermediate key,value pairs to file and read it again
>>
>> let us say i have to write each intermediate pair received @reducer to a
>> file and again read that as key value pair again and use it for processing
>>
>> I found IFile.java file which has reader and writer but i am not able
>> understand how to use it for example. I dont understand Counter value as
>> last parameter "spilledRecordsCounter"
>>
>>
>> Thanks.
>>
>>
>> --
>> *
>> *
>> *
>>
>> Regards,*
>> * Vikas *
>>
>
--
*
*
*
Regards,*
* Vikas *
Re: Writing intermediate key,value pairs to file and read it again
Posted by Vikas Jadhav <vi...@gmail.com>.
ya i looked @ chain reducer but it will not help in my case
Now i am able to write each key value pair received @ reducer(using
IFile.Writer )
but when i try to read it using IFile.Reader it reads all data not single
key value pair
Example:
*IFile.Reader reader = new* IFile.Reader<KEYIN,
VALUEIN>(context.getConfiguration(), localFs, file, *null*, myReadCounter);
DataInputBuffer keyRead = new DataInputBuffer();
DataInputBuffer valueRead = new DataInputBuffer();
reader.next(keyRead, valueRead);
when i say keyRead.getData()
it print all key value pair in that buffer i want single key value pair.
On Sun, Apr 21, 2013 at 9:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> you would look at chain reducer java doc, which meet your requirement.
>
> --Send from my Sony mobile.
> On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
>
>> Hello,
>> Can anyone help me in following issue
>> Writing intermediate key,value pairs to file and read it again
>>
>> let us say i have to write each intermediate pair received @reducer to a
>> file and again read that as key value pair again and use it for processing
>>
>> I found IFile.java file which has reader and writer but i am not able
>> understand how to use it for example. I dont understand Counter value as
>> last parameter "spilledRecordsCounter"
>>
>>
>> Thanks.
>>
>>
>> --
>> *
>> *
>> *
>>
>> Regards,*
>> * Vikas *
>>
>
--
*
*
*
Regards,*
* Vikas *
Re: Writing intermediate key,value pairs to file and read it again
Posted by Azuryy Yu <az...@gmail.com>.
you would look at chain reducer java doc, which meet your requirement.
--Send from my Sony mobile.
On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Jay Vyas <ja...@gmail.com>.
How many intermediate keys? If small enough, you can keep them in memory. If large, you can just wait for the job to finish and siphon them into your job as input with the MultipleInputs API.
On Apr 20, 2013, at 10:43 AM, Vikas Jadhav <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able understand how to use it for example. I dont understand Counter value as last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
>
>
> Regards,
> Vikas
Re: Writing intermediate key,value pairs to file and read it again
Posted by Rishi Yadav <ri...@infoobjects.com>.
writing a map only job will do the trick for you.
On Sat, Apr 20, 2013 at 8:43 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Jay Vyas <ja...@gmail.com>.
How many intermediate keys? If small enough, you can keep them in memory. If large, you can just wait for the job to finish and siphon them into your job as input with the MultipleInputs API.
On Apr 20, 2013, at 10:43 AM, Vikas Jadhav <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able understand how to use it for example. I dont understand Counter value as last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
>
>
> Regards,
> Vikas
Re: Writing intermediate key,value pairs to file and read it again
Posted by Rishi Yadav <ri...@infoobjects.com>.
writing a map only job will do the trick for you.
On Sat, Apr 20, 2013 at 8:43 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Rishi Yadav <ri...@infoobjects.com>.
writing a map only job will do the trick for you.
On Sat, Apr 20, 2013 at 8:43 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Azuryy Yu <az...@gmail.com>.
you would look at chain reducer java doc, which meet your requirement.
--Send from my Sony mobile.
On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Jay Vyas <ja...@gmail.com>.
How many intermediate keys? If small enough, you can keep them in memory. If large, you can just wait for the job to finish and siphon them into your job as input with the MultipleInputs API.
On Apr 20, 2013, at 10:43 AM, Vikas Jadhav <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able understand how to use it for example. I dont understand Counter value as last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
>
>
> Regards,
> Vikas
Re: Writing intermediate key,value pairs to file and read it again
Posted by Jay Vyas <ja...@gmail.com>.
How many intermediate keys? If small enough, you can keep them in memory. If large, you can just wait for the job to finish and siphon them into your job as input with the MultipleInputs API.
On Apr 20, 2013, at 10:43 AM, Vikas Jadhav <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able understand how to use it for example. I dont understand Counter value as last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
>
>
> Regards,
> Vikas
Re: Writing intermediate key,value pairs to file and read it again
Posted by Azuryy Yu <az...@gmail.com>.
you would look at chain reducer java doc, which meet your requirement.
--Send from my Sony mobile.
On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Azuryy Yu <az...@gmail.com>.
you would look at chain reducer java doc, which meet your requirement.
--Send from my Sony mobile.
On Apr 20, 2013 11:43 PM, "Vikas Jadhav" <vi...@gmail.com> wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>
Re: Writing intermediate key,value pairs to file and read it again
Posted by Rishi Yadav <ri...@infoobjects.com>.
writing a map only job will do the trick for you.
On Sat, Apr 20, 2013 at 8:43 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
>
> let us say i have to write each intermediate pair received @reducer to a
> file and again read that as key value pair again and use it for processing
>
> I found IFile.java file which has reader and writer but i am not able
> understand how to use it for example. I dont understand Counter value as
> last parameter "spilledRecordsCounter"
>
>
> Thanks.
>
>
> --
> *
> *
> *
>
> Regards,*
> * Vikas *
>