You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Shah, Nishant" <ni...@amazon.com> on 2013/05/29 22:50:46 UTC
Extracting status code from hbase
Hi Everyone,
I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl, I want to extract all the information from the column family 'f'.
For this I do,
Scan s = new Scan();
ResultScanner scanner = table.getScanner(s);
try {
// Scanners return Result instances.
// Now, for the actual iteration. One way is to use a while loop
// like so:
for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
// print out the row we found and the columns we were looking
// for
System.out.println("Found row: " + rr);
String[] rrs=getColumnsInColumnFamily(rr,"f");
NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
Iterator entries = familyMap.entrySet().iterator();
while(entries.hasNext()){
Entry thisEntry = (Entry) entries.next();
Object key = thisEntry.getKey();
Object val = thisEntry.getValue();
System.out.println(Bytes.toString((byte[]) key)+"="+Bytes.toString((byte[]) val));
}
The value for status is blank. It's not null, but blank. Same is the case with headers. 'mtdt' family and rest of the 'f' family is fine.
Can anyone suggest why this is happening ?
Thanks,
Nishant
RE: Extracting status code from hbase
Posted by "Shah, Nishant" <ni...@amazon.com>.
A little early to jump the gun again. Worked perfectly. I completely forgot the status codes map to 1,2 etc.
-----Original Message-----
From: Shah, Nishant
Sent: Thursday, May 30, 2013 12:25 PM
To: user@nutch.apache.org
Subject: RE: Extracting status code from hbase
I tried the below code with the Avro -1.3.3.jar included. The BinaryDecoder shows as deprecated. If I use the below code, I get the first value as 1 and second as 0. This doesn't seem to be the correct value. So I tried using the DecoderFactory as DecoderFactory.get().binaryDecoder(byteValue,null) but it says get is not defined. What do you think is going wrong here ?
Also, what other way is there to decode this if not manual. Can we use gora for this ?
Sorry if this is the wrong place to post. I could post in the avro or gora or hbase list if you guys feel that's more appropriate.
Thanks.
-----Original Message-----
From: Ferdy Galema [mailto:ferdy.galema@kalooga.com]
Sent: Thursday, May 30, 2013 1:16 AM
To: user@nutch.apache.org
Subject: Re: Extracting status code from hbase
Just to add how to manually decode ProtocolStatus:
ByteArrayInputStream bis = new ByteArrayInputStream(bytes); BinaryDecoder bd = new BinaryDecoder(bis); System.out.print(bd.readInt()); //first value is an Integer //second value is an array of for(long i = bd.readArrayStart(); i != 0; i = bd.arrayNext()) {
for (long j = 0; j < i; j++) {
System.out.print(bd.readString(null).toString());
}
}
System.out.print(bd.readLong()); //last value is a Long
On Thu, May 30, 2013 at 10:10 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> Hi,
>
> I might be able to shed some light on this.
>
> Every row in HBase webtable is a serialized
> org.apache.nutch.storage.WebPage. When you use Gora related wrappers
> (like the various Nutch jobs such as ParserJob, FetcherJob) you use
> the Avro schemas so you don't have to think about how it is encoded.
> You get a WebPage object that has all fields that you specified on the input of a Job.
>
> When you want to bypass this for some reason, you can decode the
> fields
> manually:
>
> The bytes of the columns are encoded in a specific manner. I see that
> your testcode simply tries to interpret every value as a UTF-8 encoded string.
> That is what Bytes.toString(byte[]) assumes. Although this works for
> certain fields because they actually are UTF-8 encoded strings, some
> values are encoded differently. HBaseByteInterface (in the Gora
> project) shows the different encodings. For example, the status field
> f:st "status" is an Integer (as indicated in WebPage class) that is
> encoded as
> Bytes.toBytes(int) and should be decoded accordingly with
> Bytes.toInt(byte[]). The difficult fields are f:prot "ProtocolStatus"
> and p:st "ParseStatus" are Avro records (so they have a schema of their own).
> To decode those, you can use the same code as in HBaseByteInterface.
> That is, a combination of SpecificDatumReader and BinaryDecoder.
>
> Good luck.
>
>
>
>
> On Thu, May 30, 2013 at 4:12 AM, Shah, Nishant <ni...@amazon.com> wrote:
>
>> Thanks for the reply. Will look into your suggestions.
>>
>> -----Original Message-----
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> Sent: Wednesday, May 29, 2013 7:09 PM
>> To: user@nutch.apache.org
>> Subject: Re: Extracting status code from hbase
>>
>> OH, BTW I meant to refer you to the test in line 178 of [0].
>> testPutNested hth Lewis
>>
>>
>> On Wed, May 29, 2013 at 7:07 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>> > This is most certainly better aimed at either Gora or HBase lists.
>> > Obtaining better (and consistent) understanding and of course
>> > abstracting users from such data structures is what we have been
>> > addressing in current Gora development. (See GORA-174) You will
>> > want to look specifically at some of the testing we do for this
>> > stuff over in Goran namely in [0-1].
>> > Specifically, the Query API in Gora for some data store
>> > implementations could `probably` do with some attention... so
>> > please voice you opinion over on user@gora if it tickles your fancy.
>> > Thanks
>> > Lewis
>> >
>> >
>> > [0]
>> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org
>> > /ap ache/gora/store/DataStoreTestBase.java?view=markup
>> > [1]
>> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro
>> > /we
>> > bpage.json?view=markup
>> >
>> >
>> > On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com>
>> wrote:
>> >
>> >> Hi Everyone,
>> >>
>> >> I got my error. I was trying to use toString for a field which is
>> >> int or float or long. But this leads me to another question.
>> >> The protocol status is a nested structure. Similar to parseStatus.
>> >> How could we parse these to get the individual majorcode,
>> minorcode,args ?
>> >> Also, how to detect if a url has returned a 404, or 200 or any
>> >> other status code ?
>> >> Thanks.
>> >>
>> >> -----Original Message-----
>> >> From: Shah, Nishant
>> >> Sent: Wednesday, May 29, 2013 1:51 PM
>> >> To: user@nutch.apache.org
>> >> Subject: Extracting status code from hbase
>> >>
>> >> Hi Everyone,
>> >>
>> >> I have my Nutch 2.1 setup with Hbase. Once I am done with the
>> >> crawl, I want to extract all the information from the column family 'f'.
>> >> For this I do,
>> >>
>> >> Scan s = new Scan();
>> >> ResultScanner scanner = table.getScanner(s); try { // Scanners
>> >> return Result instances.
>> >> // Now, for the actual iteration. One way is to use a while loop
>> >> // like
>> >> so:
>> >> for (Result rr = scanner.next(); rr != null; rr = scanner.next())
>> >> { // print out the row we found and the columns we were looking //
>> >> for System.out.println("Found row: " + rr); String[]
>> >> rrs=getColumnsInColumnFamily(rr,"f");
>> >> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
>> >> Iterator entries = familyMap.entrySet().iterator();
>> >> while(entries.hasNext()){
>> >>
>> >> Entry thisEntry = (Entry) entries.next(); Object key =
>> >> thisEntry.getKey(); Object val = thisEntry.getValue();
>> >> System.out.println(Bytes.toString((byte[])
>> >> key)+"="+Bytes.toString((byte[]) val)); }
>> >>
>> >> The value for status is blank. It's not null, but blank. Same is
>> >> the case with headers. 'mtdt' family and rest of the 'f' family is fine.
>> >> Can anyone suggest why this is happening ?
>> >> Thanks,
>> >> Nishant
>> >>
>> >
>> >
>> >
>> > --
>> > *Lewis*
>> >
>>
>>
>>
>> --
>> *Lewis*
>>
>
>
>
> --
> *Ferdy Galema*
> Kalooga Development
>
--
*Ferdy Galema*
Kalooga Development
--
*Kalooga* | Visual RelevanceCheck out our Visual Gallery Layer now!<http://www.independent.co.uk/arts-entertainment/music/news/david-cameron-gets-teenage-kicks-starring-in-one-direction-music-video-8499282.html#!kalooga-10369/%22One%20Direction%22>
Kalooga
Helperpark 288
9723 ZA Groningen
The Netherlands
+31 50 2103400
www.kalooga.com
info@kalooga.comKalooga EMEA
53 Davies Street
W1K 5JH London
United Kingdom
+44 20 7129 1430Kalooga Spain and LatAM
Maria de Sevilla Diago No 3
28022 Madrid - Madrid
Spain
+34 670 580 872
RE: Extracting status code from hbase
Posted by "Shah, Nishant" <ni...@amazon.com>.
I tried the below code with the Avro -1.3.3.jar included. The BinaryDecoder shows as deprecated. If I use the below code, I get the first value as 1 and second as 0. This doesn't seem to be the correct value. So I tried using the DecoderFactory as DecoderFactory.get().binaryDecoder(byteValue,null) but it says get is not defined. What do you think is going wrong here ?
Also, what other way is there to decode this if not manual. Can we use gora for this ?
Sorry if this is the wrong place to post. I could post in the avro or gora or hbase list if you guys feel that's more appropriate.
Thanks.
-----Original Message-----
From: Ferdy Galema [mailto:ferdy.galema@kalooga.com]
Sent: Thursday, May 30, 2013 1:16 AM
To: user@nutch.apache.org
Subject: Re: Extracting status code from hbase
Just to add how to manually decode ProtocolStatus:
ByteArrayInputStream bis = new ByteArrayInputStream(bytes); BinaryDecoder bd = new BinaryDecoder(bis); System.out.print(bd.readInt()); //first value is an Integer //second value is an array of for(long i = bd.readArrayStart(); i != 0; i = bd.arrayNext()) {
for (long j = 0; j < i; j++) {
System.out.print(bd.readString(null).toString());
}
}
System.out.print(bd.readLong()); //last value is a Long
On Thu, May 30, 2013 at 10:10 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> Hi,
>
> I might be able to shed some light on this.
>
> Every row in HBase webtable is a serialized
> org.apache.nutch.storage.WebPage. When you use Gora related wrappers
> (like the various Nutch jobs such as ParserJob, FetcherJob) you use
> the Avro schemas so you don't have to think about how it is encoded.
> You get a WebPage object that has all fields that you specified on the input of a Job.
>
> When you want to bypass this for some reason, you can decode the
> fields
> manually:
>
> The bytes of the columns are encoded in a specific manner. I see that
> your testcode simply tries to interpret every value as a UTF-8 encoded string.
> That is what Bytes.toString(byte[]) assumes. Although this works for
> certain fields because they actually are UTF-8 encoded strings, some
> values are encoded differently. HBaseByteInterface (in the Gora
> project) shows the different encodings. For example, the status field
> f:st "status" is an Integer (as indicated in WebPage class) that is
> encoded as
> Bytes.toBytes(int) and should be decoded accordingly with
> Bytes.toInt(byte[]). The difficult fields are f:prot "ProtocolStatus"
> and p:st "ParseStatus" are Avro records (so they have a schema of their own).
> To decode those, you can use the same code as in HBaseByteInterface.
> That is, a combination of SpecificDatumReader and BinaryDecoder.
>
> Good luck.
>
>
>
>
> On Thu, May 30, 2013 at 4:12 AM, Shah, Nishant <ni...@amazon.com> wrote:
>
>> Thanks for the reply. Will look into your suggestions.
>>
>> -----Original Message-----
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> Sent: Wednesday, May 29, 2013 7:09 PM
>> To: user@nutch.apache.org
>> Subject: Re: Extracting status code from hbase
>>
>> OH, BTW I meant to refer you to the test in line 178 of [0].
>> testPutNested hth Lewis
>>
>>
>> On Wed, May 29, 2013 at 7:07 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>> > This is most certainly better aimed at either Gora or HBase lists.
>> > Obtaining better (and consistent) understanding and of course
>> > abstracting users from such data structures is what we have been
>> > addressing in current Gora development. (See GORA-174) You will
>> > want to look specifically at some of the testing we do for this
>> > stuff over in Goran namely in [0-1].
>> > Specifically, the Query API in Gora for some data store
>> > implementations could `probably` do with some attention... so
>> > please voice you opinion over on user@gora if it tickles your fancy.
>> > Thanks
>> > Lewis
>> >
>> >
>> > [0]
>> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org
>> > /ap ache/gora/store/DataStoreTestBase.java?view=markup
>> > [1]
>> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro
>> > /we
>> > bpage.json?view=markup
>> >
>> >
>> > On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com>
>> wrote:
>> >
>> >> Hi Everyone,
>> >>
>> >> I got my error. I was trying to use toString for a field which is
>> >> int or float or long. But this leads me to another question.
>> >> The protocol status is a nested structure. Similar to parseStatus.
>> >> How could we parse these to get the individual majorcode,
>> minorcode,args ?
>> >> Also, how to detect if a url has returned a 404, or 200 or any
>> >> other status code ?
>> >> Thanks.
>> >>
>> >> -----Original Message-----
>> >> From: Shah, Nishant
>> >> Sent: Wednesday, May 29, 2013 1:51 PM
>> >> To: user@nutch.apache.org
>> >> Subject: Extracting status code from hbase
>> >>
>> >> Hi Everyone,
>> >>
>> >> I have my Nutch 2.1 setup with Hbase. Once I am done with the
>> >> crawl, I want to extract all the information from the column family 'f'.
>> >> For this I do,
>> >>
>> >> Scan s = new Scan();
>> >> ResultScanner scanner = table.getScanner(s); try { // Scanners
>> >> return Result instances.
>> >> // Now, for the actual iteration. One way is to use a while loop
>> >> // like
>> >> so:
>> >> for (Result rr = scanner.next(); rr != null; rr = scanner.next())
>> >> { // print out the row we found and the columns we were looking //
>> >> for System.out.println("Found row: " + rr); String[]
>> >> rrs=getColumnsInColumnFamily(rr,"f");
>> >> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
>> >> Iterator entries = familyMap.entrySet().iterator();
>> >> while(entries.hasNext()){
>> >>
>> >> Entry thisEntry = (Entry) entries.next(); Object key =
>> >> thisEntry.getKey(); Object val = thisEntry.getValue();
>> >> System.out.println(Bytes.toString((byte[])
>> >> key)+"="+Bytes.toString((byte[]) val)); }
>> >>
>> >> The value for status is blank. It's not null, but blank. Same is
>> >> the case with headers. 'mtdt' family and rest of the 'f' family is fine.
>> >> Can anyone suggest why this is happening ?
>> >> Thanks,
>> >> Nishant
>> >>
>> >
>> >
>> >
>> > --
>> > *Lewis*
>> >
>>
>>
>>
>> --
>> *Lewis*
>>
>
>
>
> --
> *Ferdy Galema*
> Kalooga Development
>
--
*Ferdy Galema*
Kalooga Development
--
*Kalooga* | Visual RelevanceCheck out our Visual Gallery Layer now!<http://www.independent.co.uk/arts-entertainment/music/news/david-cameron-gets-teenage-kicks-starring-in-one-direction-music-video-8499282.html#!kalooga-10369/%22One%20Direction%22>
Kalooga
Helperpark 288
9723 ZA Groningen
The Netherlands
+31 50 2103400
www.kalooga.com
info@kalooga.comKalooga EMEA
53 Davies Street
W1K 5JH London
United Kingdom
+44 20 7129 1430Kalooga Spain and LatAM
Maria de Sevilla Diago No 3
28022 Madrid - Madrid
Spain
+34 670 580 872
Re: Extracting status code from hbase
Posted by Ferdy Galema <fe...@kalooga.com>.
Just to add how to manually decode ProtocolStatus:
ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
BinaryDecoder bd = new BinaryDecoder(bis);
System.out.print(bd.readInt()); //first value is an Integer
//second value is an array of
for(long i = bd.readArrayStart(); i != 0; i = bd.arrayNext()) {
for (long j = 0; j < i; j++) {
System.out.print(bd.readString(null).toString());
}
}
System.out.print(bd.readLong()); //last value is a Long
On Thu, May 30, 2013 at 10:10 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> Hi,
>
> I might be able to shed some light on this.
>
> Every row in HBase webtable is a serialized
> org.apache.nutch.storage.WebPage. When you use Gora related wrappers (like
> the various Nutch jobs such as ParserJob, FetcherJob) you use the Avro
> schemas so you don't have to think about how it is encoded. You get a
> WebPage object that has all fields that you specified on the input of a Job.
>
> When you want to bypass this for some reason, you can decode the fields
> manually:
>
> The bytes of the columns are encoded in a specific manner. I see that your
> testcode simply tries to interpret every value as a UTF-8 encoded string.
> That is what Bytes.toString(byte[]) assumes. Although this works for
> certain fields because they actually are UTF-8 encoded strings, some values
> are encoded differently. HBaseByteInterface (in the Gora project) shows the
> different encodings. For example, the status field f:st "status" is an
> Integer (as indicated in WebPage class) that is encoded as
> Bytes.toBytes(int) and should be decoded accordingly with
> Bytes.toInt(byte[]). The difficult fields are f:prot "ProtocolStatus" and
> p:st "ParseStatus" are Avro records (so they have a schema of their own).
> To decode those, you can use the same code as in HBaseByteInterface. That
> is, a combination of SpecificDatumReader and BinaryDecoder.
>
> Good luck.
>
>
>
>
> On Thu, May 30, 2013 at 4:12 AM, Shah, Nishant <ni...@amazon.com> wrote:
>
>> Thanks for the reply. Will look into your suggestions.
>>
>> -----Original Message-----
>> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
>> Sent: Wednesday, May 29, 2013 7:09 PM
>> To: user@nutch.apache.org
>> Subject: Re: Extracting status code from hbase
>>
>> OH, BTW I meant to refer you to the test in line 178 of [0].
>> testPutNested hth Lewis
>>
>>
>> On Wed, May 29, 2013 at 7:07 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>> > This is most certainly better aimed at either Gora or HBase lists.
>> > Obtaining better (and consistent) understanding and of course
>> > abstracting users from such data structures is what we have been
>> > addressing in current Gora development. (See GORA-174) You will want
>> > to look specifically at some of the testing we do for this stuff over
>> > in Goran namely in [0-1].
>> > Specifically, the Query API in Gora for some data store
>> > implementations could `probably` do with some attention... so please
>> > voice you opinion over on user@gora if it tickles your fancy.
>> > Thanks
>> > Lewis
>> >
>> >
>> > [0]
>> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org/ap
>> > ache/gora/store/DataStoreTestBase.java?view=markup
>> > [1]
>> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro/we
>> > bpage.json?view=markup
>> >
>> >
>> > On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com>
>> wrote:
>> >
>> >> Hi Everyone,
>> >>
>> >> I got my error. I was trying to use toString for a field which is int
>> >> or float or long. But this leads me to another question.
>> >> The protocol status is a nested structure. Similar to parseStatus.
>> >> How could we parse these to get the individual majorcode,
>> minorcode,args ?
>> >> Also, how to detect if a url has returned a 404, or 200 or any other
>> >> status code ?
>> >> Thanks.
>> >>
>> >> -----Original Message-----
>> >> From: Shah, Nishant
>> >> Sent: Wednesday, May 29, 2013 1:51 PM
>> >> To: user@nutch.apache.org
>> >> Subject: Extracting status code from hbase
>> >>
>> >> Hi Everyone,
>> >>
>> >> I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl,
>> >> I want to extract all the information from the column family 'f'.
>> >> For this I do,
>> >>
>> >> Scan s = new Scan();
>> >> ResultScanner scanner = table.getScanner(s); try { // Scanners return
>> >> Result instances.
>> >> // Now, for the actual iteration. One way is to use a while loop //
>> >> like
>> >> so:
>> >> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
>> >> // print out the row we found and the columns we were looking // for
>> >> System.out.println("Found row: " + rr); String[]
>> >> rrs=getColumnsInColumnFamily(rr,"f");
>> >> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
>> >> Iterator entries = familyMap.entrySet().iterator();
>> >> while(entries.hasNext()){
>> >>
>> >> Entry thisEntry = (Entry) entries.next(); Object key =
>> >> thisEntry.getKey(); Object val = thisEntry.getValue();
>> >> System.out.println(Bytes.toString((byte[])
>> >> key)+"="+Bytes.toString((byte[]) val)); }
>> >>
>> >> The value for status is blank. It's not null, but blank. Same is the
>> >> case with headers. 'mtdt' family and rest of the 'f' family is fine.
>> >> Can anyone suggest why this is happening ?
>> >> Thanks,
>> >> Nishant
>> >>
>> >
>> >
>> >
>> > --
>> > *Lewis*
>> >
>>
>>
>>
>> --
>> *Lewis*
>>
>
>
>
> --
> *Ferdy Galema*
> Kalooga Development
>
--
*Ferdy Galema*
Kalooga Development
--
*Kalooga* | Visual RelevanceCheck out our Visual Gallery Layer now!<http://www.independent.co.uk/arts-entertainment/music/news/david-cameron-gets-teenage-kicks-starring-in-one-direction-music-video-8499282.html#!kalooga-10369/%22One%20Direction%22>
Kalooga
Helperpark 288
9723 ZA Groningen
The Netherlands
+31 50 2103400
www.kalooga.com
info@kalooga.comKalooga EMEA
53 Davies Street
W1K 5JH London
United Kingdom
+44 20 7129 1430Kalooga Spain and LatAM
Maria de Sevilla Diago No 3
28022 Madrid - Madrid
Spain
+34 670 580 872
Re: Extracting status code from hbase
Posted by Ferdy Galema <fe...@kalooga.com>.
Hi,
I might be able to shed some light on this.
Every row in HBase webtable is a serialized
org.apache.nutch.storage.WebPage. When you use Gora related wrappers (like
the various Nutch jobs such as ParserJob, FetcherJob) you use the Avro
schemas so you don't have to think about how it is encoded. You get a
WebPage object that has all fields that you specified on the input of a Job.
When you want to bypass this for some reason, you can decode the fields
manually:
The bytes of the columns are encoded in a specific manner. I see that your
testcode simply tries to interpret every value as a UTF-8 encoded string.
That is what Bytes.toString(byte[]) assumes. Although this works for
certain fields because they actually are UTF-8 encoded strings, some values
are encoded differently. HBaseByteInterface (in the Gora project) shows the
different encodings. For example, the status field f:st "status" is an
Integer (as indicated in WebPage class) that is encoded as
Bytes.toBytes(int) and should be decoded accordingly with
Bytes.toInt(byte[]). The difficult fields are f:prot "ProtocolStatus" and
p:st "ParseStatus" are Avro records (so they have a schema of their own).
To decode those, you can use the same code as in HBaseByteInterface. That
is, a combination of SpecificDatumReader and BinaryDecoder.
Good luck.
On Thu, May 30, 2013 at 4:12 AM, Shah, Nishant <ni...@amazon.com> wrote:
> Thanks for the reply. Will look into your suggestions.
>
> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Wednesday, May 29, 2013 7:09 PM
> To: user@nutch.apache.org
> Subject: Re: Extracting status code from hbase
>
> OH, BTW I meant to refer you to the test in line 178 of [0]. testPutNested
> hth Lewis
>
>
> On Wed, May 29, 2013 at 7:07 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
> > This is most certainly better aimed at either Gora or HBase lists.
> > Obtaining better (and consistent) understanding and of course
> > abstracting users from such data structures is what we have been
> > addressing in current Gora development. (See GORA-174) You will want
> > to look specifically at some of the testing we do for this stuff over
> > in Goran namely in [0-1].
> > Specifically, the Query API in Gora for some data store
> > implementations could `probably` do with some attention... so please
> > voice you opinion over on user@gora if it tickles your fancy.
> > Thanks
> > Lewis
> >
> >
> > [0]
> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org/ap
> > ache/gora/store/DataStoreTestBase.java?view=markup
> > [1]
> > http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro/we
> > bpage.json?view=markup
> >
> >
> > On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com>
> wrote:
> >
> >> Hi Everyone,
> >>
> >> I got my error. I was trying to use toString for a field which is int
> >> or float or long. But this leads me to another question.
> >> The protocol status is a nested structure. Similar to parseStatus.
> >> How could we parse these to get the individual majorcode,
> minorcode,args ?
> >> Also, how to detect if a url has returned a 404, or 200 or any other
> >> status code ?
> >> Thanks.
> >>
> >> -----Original Message-----
> >> From: Shah, Nishant
> >> Sent: Wednesday, May 29, 2013 1:51 PM
> >> To: user@nutch.apache.org
> >> Subject: Extracting status code from hbase
> >>
> >> Hi Everyone,
> >>
> >> I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl,
> >> I want to extract all the information from the column family 'f'.
> >> For this I do,
> >>
> >> Scan s = new Scan();
> >> ResultScanner scanner = table.getScanner(s); try { // Scanners return
> >> Result instances.
> >> // Now, for the actual iteration. One way is to use a while loop //
> >> like
> >> so:
> >> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
> >> // print out the row we found and the columns we were looking // for
> >> System.out.println("Found row: " + rr); String[]
> >> rrs=getColumnsInColumnFamily(rr,"f");
> >> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
> >> Iterator entries = familyMap.entrySet().iterator();
> >> while(entries.hasNext()){
> >>
> >> Entry thisEntry = (Entry) entries.next(); Object key =
> >> thisEntry.getKey(); Object val = thisEntry.getValue();
> >> System.out.println(Bytes.toString((byte[])
> >> key)+"="+Bytes.toString((byte[]) val)); }
> >>
> >> The value for status is blank. It's not null, but blank. Same is the
> >> case with headers. 'mtdt' family and rest of the 'f' family is fine.
> >> Can anyone suggest why this is happening ?
> >> Thanks,
> >> Nishant
> >>
> >
> >
> >
> > --
> > *Lewis*
> >
>
>
>
> --
> *Lewis*
>
--
*Ferdy Galema*
Kalooga Development
--
*Kalooga* | Visual RelevanceCheck out our Visual Gallery Layer now!<http://www.independent.co.uk/arts-entertainment/music/news/david-cameron-gets-teenage-kicks-starring-in-one-direction-music-video-8499282.html#!kalooga-10369/%22One%20Direction%22>
Kalooga
Helperpark 288
9723 ZA Groningen
The Netherlands
+31 50 2103400
www.kalooga.com
info@kalooga.comKalooga EMEA
53 Davies Street
W1K 5JH London
United Kingdom
+44 20 7129 1430Kalooga Spain and LatAM
Maria de Sevilla Diago No 3
28022 Madrid - Madrid
Spain
+34 670 580 872
RE: Extracting status code from hbase
Posted by "Shah, Nishant" <ni...@amazon.com>.
Thanks for the reply. Will look into your suggestions.
-----Original Message-----
From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
Sent: Wednesday, May 29, 2013 7:09 PM
To: user@nutch.apache.org
Subject: Re: Extracting status code from hbase
OH, BTW I meant to refer you to the test in line 178 of [0]. testPutNested hth Lewis
On Wed, May 29, 2013 at 7:07 PM, Lewis John Mcgibbney < lewis.mcgibbney@gmail.com> wrote:
> This is most certainly better aimed at either Gora or HBase lists.
> Obtaining better (and consistent) understanding and of course
> abstracting users from such data structures is what we have been
> addressing in current Gora development. (See GORA-174) You will want
> to look specifically at some of the testing we do for this stuff over
> in Goran namely in [0-1].
> Specifically, the Query API in Gora for some data store
> implementations could `probably` do with some attention... so please
> voice you opinion over on user@gora if it tickles your fancy.
> Thanks
> Lewis
>
>
> [0]
> http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org/ap
> ache/gora/store/DataStoreTestBase.java?view=markup
> [1]
> http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro/we
> bpage.json?view=markup
>
>
> On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com> wrote:
>
>> Hi Everyone,
>>
>> I got my error. I was trying to use toString for a field which is int
>> or float or long. But this leads me to another question.
>> The protocol status is a nested structure. Similar to parseStatus.
>> How could we parse these to get the individual majorcode, minorcode,args ?
>> Also, how to detect if a url has returned a 404, or 200 or any other
>> status code ?
>> Thanks.
>>
>> -----Original Message-----
>> From: Shah, Nishant
>> Sent: Wednesday, May 29, 2013 1:51 PM
>> To: user@nutch.apache.org
>> Subject: Extracting status code from hbase
>>
>> Hi Everyone,
>>
>> I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl,
>> I want to extract all the information from the column family 'f'.
>> For this I do,
>>
>> Scan s = new Scan();
>> ResultScanner scanner = table.getScanner(s); try { // Scanners return
>> Result instances.
>> // Now, for the actual iteration. One way is to use a while loop //
>> like
>> so:
>> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
>> // print out the row we found and the columns we were looking // for
>> System.out.println("Found row: " + rr); String[]
>> rrs=getColumnsInColumnFamily(rr,"f");
>> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
>> Iterator entries = familyMap.entrySet().iterator();
>> while(entries.hasNext()){
>>
>> Entry thisEntry = (Entry) entries.next(); Object key =
>> thisEntry.getKey(); Object val = thisEntry.getValue();
>> System.out.println(Bytes.toString((byte[])
>> key)+"="+Bytes.toString((byte[]) val)); }
>>
>> The value for status is blank. It's not null, but blank. Same is the
>> case with headers. 'mtdt' family and rest of the 'f' family is fine.
>> Can anyone suggest why this is happening ?
>> Thanks,
>> Nishant
>>
>
>
>
> --
> *Lewis*
>
--
*Lewis*
Re: Extracting status code from hbase
Posted by Lewis John Mcgibbney <le...@gmail.com>.
OH, BTW I meant to refer you to the test in line 178 of [0]. testPutNested
hth
Lewis
On Wed, May 29, 2013 at 7:07 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> This is most certainly better aimed at either Gora or HBase lists.
> Obtaining better (and consistent) understanding and of course abstracting
> users from such data structures is what we have been addressing in current
> Gora development. (See GORA-174)
> You will want to look specifically at some of the testing we do for this
> stuff over in Goran namely in [0-1].
> Specifically, the Query API in Gora for some data store implementations
> could `probably` do with some attention... so please voice you opinion over
> on user@gora if it tickles your fancy.
> Thanks
> Lewis
>
>
> [0]
> http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java?view=markup
> [1]
> http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro/webpage.json?view=markup
>
>
> On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com> wrote:
>
>> Hi Everyone,
>>
>> I got my error. I was trying to use toString for a field which is int or
>> float or long. But this leads me to another question.
>> The protocol status is a nested structure. Similar to parseStatus. How
>> could we parse these to get the individual majorcode, minorcode,args ?
>> Also, how to detect if a url has returned a 404, or 200 or any other
>> status code ?
>> Thanks.
>>
>> -----Original Message-----
>> From: Shah, Nishant
>> Sent: Wednesday, May 29, 2013 1:51 PM
>> To: user@nutch.apache.org
>> Subject: Extracting status code from hbase
>>
>> Hi Everyone,
>>
>> I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl, I
>> want to extract all the information from the column family 'f'.
>> For this I do,
>>
>> Scan s = new Scan();
>> ResultScanner scanner = table.getScanner(s); try { // Scanners return
>> Result instances.
>> // Now, for the actual iteration. One way is to use a while loop // like
>> so:
>> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) { //
>> print out the row we found and the columns we were looking // for
>> System.out.println("Found row: " + rr); String[]
>> rrs=getColumnsInColumnFamily(rr,"f");
>> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
>> Iterator entries = familyMap.entrySet().iterator();
>> while(entries.hasNext()){
>>
>> Entry thisEntry = (Entry) entries.next(); Object key =
>> thisEntry.getKey(); Object val = thisEntry.getValue();
>> System.out.println(Bytes.toString((byte[])
>> key)+"="+Bytes.toString((byte[]) val)); }
>>
>> The value for status is blank. It's not null, but blank. Same is the case
>> with headers. 'mtdt' family and rest of the 'f' family is fine.
>> Can anyone suggest why this is happening ?
>> Thanks,
>> Nishant
>>
>
>
>
> --
> *Lewis*
>
--
*Lewis*
Re: Extracting status code from hbase
Posted by Lewis John Mcgibbney <le...@gmail.com>.
This is most certainly better aimed at either Gora or HBase lists.
Obtaining better (and consistent) understanding and of course abstracting
users from such data structures is what we have been addressing in current
Gora development. (See GORA-174)
You will want to look specifically at some of the testing we do for this
stuff over in Goran namely in [0-1].
Specifically, the Query API in Gora for some data store implementations
could `probably` do with some attention... so please voice you opinion over
on user@gora if it tickles your fancy.
Thanks
Lewis
[0]
http://svn.apache.org/viewvc/gora/trunk/gora-core/src/test/java/org/apache/gora/store/DataStoreTestBase.java?view=markup
[1]
http://svn.apache.org/viewvc/gora/trunk/gora-core/src/examples/avro/webpage.json?view=markup
On Wed, May 29, 2013 at 3:55 PM, Shah, Nishant <ni...@amazon.com> wrote:
> Hi Everyone,
>
> I got my error. I was trying to use toString for a field which is int or
> float or long. But this leads me to another question.
> The protocol status is a nested structure. Similar to parseStatus. How
> could we parse these to get the individual majorcode, minorcode,args ?
> Also, how to detect if a url has returned a 404, or 200 or any other
> status code ?
> Thanks.
>
> -----Original Message-----
> From: Shah, Nishant
> Sent: Wednesday, May 29, 2013 1:51 PM
> To: user@nutch.apache.org
> Subject: Extracting status code from hbase
>
> Hi Everyone,
>
> I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl, I
> want to extract all the information from the column family 'f'.
> For this I do,
>
> Scan s = new Scan();
> ResultScanner scanner = table.getScanner(s); try { // Scanners return
> Result instances.
> // Now, for the actual iteration. One way is to use a while loop // like
> so:
> for (Result rr = scanner.next(); rr != null; rr = scanner.next()) { //
> print out the row we found and the columns we were looking // for
> System.out.println("Found row: " + rr); String[]
> rrs=getColumnsInColumnFamily(rr,"f");
> NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
> Iterator entries = familyMap.entrySet().iterator();
> while(entries.hasNext()){
>
> Entry thisEntry = (Entry) entries.next(); Object key = thisEntry.getKey();
> Object val = thisEntry.getValue();
> System.out.println(Bytes.toString((byte[])
> key)+"="+Bytes.toString((byte[]) val)); }
>
> The value for status is blank. It's not null, but blank. Same is the case
> with headers. 'mtdt' family and rest of the 'f' family is fine.
> Can anyone suggest why this is happening ?
> Thanks,
> Nishant
>
--
*Lewis*
RE: Extracting status code from hbase
Posted by "Shah, Nishant" <ni...@amazon.com>.
Hi Everyone,
I got my error. I was trying to use toString for a field which is int or float or long. But this leads me to another question.
The protocol status is a nested structure. Similar to parseStatus. How could we parse these to get the individual majorcode, minorcode,args ?
Also, how to detect if a url has returned a 404, or 200 or any other status code ?
Thanks.
-----Original Message-----
From: Shah, Nishant
Sent: Wednesday, May 29, 2013 1:51 PM
To: user@nutch.apache.org
Subject: Extracting status code from hbase
Hi Everyone,
I have my Nutch 2.1 setup with Hbase. Once I am done with the crawl, I want to extract all the information from the column family 'f'.
For this I do,
Scan s = new Scan();
ResultScanner scanner = table.getScanner(s); try { // Scanners return Result instances.
// Now, for the actual iteration. One way is to use a while loop // like so:
for (Result rr = scanner.next(); rr != null; rr = scanner.next()) { // print out the row we found and the columns we were looking // for System.out.println("Found row: " + rr); String[] rrs=getColumnsInColumnFamily(rr,"f");
NavigableMap familyMap = rr.getFamilyMap(Bytes.toBytes("f"));
Iterator entries = familyMap.entrySet().iterator(); while(entries.hasNext()){
Entry thisEntry = (Entry) entries.next(); Object key = thisEntry.getKey(); Object val = thisEntry.getValue();
System.out.println(Bytes.toString((byte[]) key)+"="+Bytes.toString((byte[]) val)); }
The value for status is blank. It's not null, but blank. Same is the case with headers. 'mtdt' family and rest of the 'f' family is fine.
Can anyone suggest why this is happening ?
Thanks,
Nishant