You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Kleegrewe, Christian" <ch...@siemens.com> on 2011/05/25 11:18:12 UTC
Mapreduce log question
Hi all,
I would like to figure out on which table region is used by a specific map task. Is there a possibility to find such informaiton in the hbase logs?
Thanks in advance
Christian
------------------8<--------------------------
Siemens AG
Corporate Technology
Corporate Research and Technologies
CT T DE IT3
Otto-Hahn-Ring 6
81739 München, Deutschland
Tel.: +49 (89) 636-42722
Fax: +49 (89) 636-41423
mailto:christian.kleegrewe@siemens.com
Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
RE: Mapreduce log question
Posted by "Kleegrewe, Christian" <ch...@siemens.com>.
Hi Dave,
Yesterday I was a little confused about the question wether we use TableInputFormat for our map reduce job and I answered no. After a more detailed review I found out that we are surely using it because we use the TableMapReduceUtil.initTableMapperJob (...) function for the map job initialization. It was a little bit tricky to figure it out and I had to have a deeper look into the sourcecode until I figured out how the TableInputFormat class is applied. Finally I only had to change the log level in the master ui and now I am running a new test with hopefully the desidred information in the logfiles.
Thanks again for your help
Christian
------------------8<--------------------------
Siemens AG
Corporate Technology
Corporate Research and Technologies
CT T DE IT3
Otto-Hahn-Ring 6
81739 München, Deutschland
Tel.: +49 (89) 636-42722
Fax: +49 (89) 636-41423
mailto:christian.kleegrewe@siemens.com
Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
-----Original Message-----
From: ddlatham@gmail.com [mailto:ddlatham@gmail.com] On Behalf Of Dave Latham
Sent: Wednesday, May 25, 2011 3:19 PM
To: user@hbase.apache.org
Subject: Re: Mapreduce log question
Are you using TableInputFormat? If so, if you turn on DEBUG level logging
for hbase (or just org.apache.hadoop.hbase.mapreduce.TableInputFormatBase)
you should see lines like this, giving the map task number, region location,
start row, and end row:
getSplits: split -> 0 ->
hslave107:,@G\xA0\xFB\xBC\x94\x8C\x00~\xFF\xE3\xD3\xEA#\xBC\xE1\xDD\xE0|6\x01\x81\xC6\x1C
getSplits: split -> 1 -> hslave117:@G
\xA0\xFB\xBC\x94\x8C\x00~\xFF\xE3\xD3\xEA#\xBC\xE1\xDD\xE0|6\x01\x81\xC6\x1C,\x80F\xB8\xD2$\xAA\xCD(\x09\xB1\xB4\xA1\xB0A\x937tY\x99r\x01\x83\xB3\x18
getSplits: split -> 2 ->
hslave94:\x80F\xB8\xD2$\xAA\xCD(\x09\xB1\xB4\xA1\xB0A\x937tY\x99r\x01\x83\xB3\x18,\xC03\xBE\xF9K\xEB\xD4\xDDR5\x8C\xECa\x98}`\x1E,\xA9\xA0\x01\x83\xB3\x18
getSplits: split -> 3 ->
hslave1:\xC03\xBE\xF9K\xEB\xD4\xDDR5\x8C\xECa\x98}`\x1E,\xA9\xA0\x01\x83\xB3\x18,
: job_201105191622_1815
Dave
On Wed, May 25, 2011 at 2:18 AM, Kleegrewe, Christian <
christian.kleegrewe@siemens.com> wrote:
> Hi all,
>
> I would like to figure out on which table region is used by a specific map
> task. Is there a possibility to find such informaiton in the hbase logs?
>
> Thanks in advance
>
> Christian
>
> ------------------8<--------------------------
>
> Siemens AG
> Corporate Technology
> Corporate Research and Technologies
> CT T DE IT3
> Otto-Hahn-Ring 6
> 81739 München, Deutschland
> Tel.: +49 (89) 636-42722
> Fax: +49 (89) 636-41423
> mailto:christian.kleegrewe@siemens.com
>
> Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme;
> Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus
> Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm,
> Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München,
> Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB
> 6684; WEEE-Reg.-Nr. DE 23691322
>
>
>
>
RE: Mapreduce log question
Posted by "Kleegrewe, Christian" <ch...@siemens.com>.
Hi Dave,
Thanks for the reply. Actually we are not using TableInputFormat. I will have a look at the class and if it makes logfiles more usable I will use it,
Best regards,
Christian
------------------8<--------------------------
Siemens AG
Corporate Technology
Corporate Research and Technologies
CT T DE IT3
Otto-Hahn-Ring 6
81739 München, Deutschland
Tel.: +49 (89) 636-42722
Fax: +49 (89) 636-41423
mailto:christian.kleegrewe@siemens.com
Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322
-----Original Message-----
From: ddlatham@gmail.com [mailto:ddlatham@gmail.com] On Behalf Of Dave Latham
Sent: Wednesday, May 25, 2011 3:19 PM
To: user@hbase.apache.org
Subject: Re: Mapreduce log question
Are you using TableInputFormat? If so, if you turn on DEBUG level logging
for hbase (or just org.apache.hadoop.hbase.mapreduce.TableInputFormatBase)
you should see lines like this, giving the map task number, region location,
start row, and end row:
getSplits: split -> 0 ->
hslave107:,@G\xA0\xFB\xBC\x94\x8C\x00~\xFF\xE3\xD3\xEA#\xBC\xE1\xDD\xE0|6\x01\x81\xC6\x1C
getSplits: split -> 1 -> hslave117:@G
\xA0\xFB\xBC\x94\x8C\x00~\xFF\xE3\xD3\xEA#\xBC\xE1\xDD\xE0|6\x01\x81\xC6\x1C,\x80F\xB8\xD2$\xAA\xCD(\x09\xB1\xB4\xA1\xB0A\x937tY\x99r\x01\x83\xB3\x18
getSplits: split -> 2 ->
hslave94:\x80F\xB8\xD2$\xAA\xCD(\x09\xB1\xB4\xA1\xB0A\x937tY\x99r\x01\x83\xB3\x18,\xC03\xBE\xF9K\xEB\xD4\xDDR5\x8C\xECa\x98}`\x1E,\xA9\xA0\x01\x83\xB3\x18
getSplits: split -> 3 ->
hslave1:\xC03\xBE\xF9K\xEB\xD4\xDDR5\x8C\xECa\x98}`\x1E,\xA9\xA0\x01\x83\xB3\x18,
: job_201105191622_1815
Dave
On Wed, May 25, 2011 at 2:18 AM, Kleegrewe, Christian <
christian.kleegrewe@siemens.com> wrote:
> Hi all,
>
> I would like to figure out on which table region is used by a specific map
> task. Is there a possibility to find such informaiton in the hbase logs?
>
> Thanks in advance
>
> Christian
>
> ------------------8<--------------------------
>
> Siemens AG
> Corporate Technology
> Corporate Research and Technologies
> CT T DE IT3
> Otto-Hahn-Ring 6
> 81739 München, Deutschland
> Tel.: +49 (89) 636-42722
> Fax: +49 (89) 636-41423
> mailto:christian.kleegrewe@siemens.com
>
> Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme;
> Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus
> Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm,
> Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München,
> Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB
> 6684; WEEE-Reg.-Nr. DE 23691322
>
>
>
>
Re: Mapreduce log question
Posted by Dave Latham <la...@davelink.net>.
Are you using TableInputFormat? If so, if you turn on DEBUG level logging
for hbase (or just org.apache.hadoop.hbase.mapreduce.TableInputFormatBase)
you should see lines like this, giving the map task number, region location,
start row, and end row:
getSplits: split -> 0 ->
hslave107:,@G\xA0\xFB\xBC\x94\x8C\x00~\xFF\xE3\xD3\xEA#\xBC\xE1\xDD\xE0|6\x01\x81\xC6\x1C
getSplits: split -> 1 -> hslave117:@G
\xA0\xFB\xBC\x94\x8C\x00~\xFF\xE3\xD3\xEA#\xBC\xE1\xDD\xE0|6\x01\x81\xC6\x1C,\x80F\xB8\xD2$\xAA\xCD(\x09\xB1\xB4\xA1\xB0A\x937tY\x99r\x01\x83\xB3\x18
getSplits: split -> 2 ->
hslave94:\x80F\xB8\xD2$\xAA\xCD(\x09\xB1\xB4\xA1\xB0A\x937tY\x99r\x01\x83\xB3\x18,\xC03\xBE\xF9K\xEB\xD4\xDDR5\x8C\xECa\x98}`\x1E,\xA9\xA0\x01\x83\xB3\x18
getSplits: split -> 3 ->
hslave1:\xC03\xBE\xF9K\xEB\xD4\xDDR5\x8C\xECa\x98}`\x1E,\xA9\xA0\x01\x83\xB3\x18,
: job_201105191622_1815
Dave
On Wed, May 25, 2011 at 2:18 AM, Kleegrewe, Christian <
christian.kleegrewe@siemens.com> wrote:
> Hi all,
>
> I would like to figure out on which table region is used by a specific map
> task. Is there a possibility to find such informaiton in the hbase logs?
>
> Thanks in advance
>
> Christian
>
> ------------------8<--------------------------
>
> Siemens AG
> Corporate Technology
> Corporate Research and Technologies
> CT T DE IT3
> Otto-Hahn-Ring 6
> 81739 München, Deutschland
> Tel.: +49 (89) 636-42722
> Fax: +49 (89) 636-41423
> mailto:christian.kleegrewe@siemens.com
>
> Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme;
> Vorstand: Peter Löscher, Vorsitzender; Roland Busch, Brigitte Ederer, Klaus
> Helmrich, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm,
> Peter Y. Solmssen, Michael Süß; Sitz der Gesellschaft: Berlin und München,
> Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB
> 6684; WEEE-Reg.-Nr. DE 23691322
>
>
>
>