You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by RS <ti...@163.com> on 2013/06/13 15:40:45 UTC
what is stored in the hbase after inject job
I do not what is sotred in the hbase after inject a website.
When I use the hbase shell $ scan 'webpage' , there are :
hbase(main):028:0> scan '1_webpage'
ROW COLUMN+CELL
com.xinhuanet.www:http/ column=f:fi, timestamp=1371110099941, value=\x00'\x8D\x00
com.xinhuanet.www:http/ column=f:ts, timestamp=1371110099941, value=\x00\x00\x01?<\x87\xBA\x0A
com.xinhuanet.www:http/ column=mk:_injmrk_, timestamp=1371110099941, value=y
com.xinhuanet.www:http/ column=mk:dist, timestamp=1371110099941, value=0
com.xinhuanet.www:http/ column=mtdt:_csh_, timestamp=1371110099941, value=?\x80\x00\x00
com.xinhuanet.www:http/ column=s:s, timestamp=1371110099941, value=?\x80\x00\x00
1 row(s) in 0.0300 seconds
So, is only 6 column are setted in the hbase ? And what is the real data stored in it?
I find that in the source code, there is a WebPage Class. I could not understand all, but I think there should be 24 fileds in the hbase for each webside.
public static final String[] _ALL_FIELDS = {"baseUrl","status","fetchTime","prevFetchTime","fetchInterval","retriesSinceFetch","modifiedTime","prevModifiedTime","protocolStatus","content","contentType","prevSignature","signature","title","text","parseStatus","score","reprUrl","headers","outlinks","inlinks","markers","metadata","batchId",};
Thanks
HeChuan
Re:Re: what is stored in the hbase after inject job
Posted by RS <ti...@163.com>.
Hello
The column=s:s is actully the score, you could find this in the gora-hbase-mapping.xml file under the conf dir.
Lewis John Mcgibbney has replyed in this mail list.
Thanks
HeChuan
At 2013-06-18 22:52:06,"Ahmet Emre Aladağ" <em...@agmlab.com> wrote:
>Hi Tejas,
>
>What about the scores? Where are they stored? I could not find.
>
>On 06/13/2013 09:00 PM, Tejas Patil wrote:
>> row-key = url
>>
>> column=f:fi : fetchInterval (the delay between re-fetches of a
>> page)
>> column=f:ts : fetchTime (indicates when the url will be elligible
>> for fetching)
>> column=mk:_injmrk_ : markers
>> column=mk:dist
>> column=mtdt:_csh_ : metadata
>> column=s:s : status (is the url fetched, unfetched, newly
>> injected, gone, redirected etc..)
>>
>>
>>
>> On Thu, Jun 13, 2013 at 6:40 AM, RS <ti...@163.com> wrote:
>>
>>> I do not what is sotred in the hbase after inject a website.
>>> When I use the hbase shell $ scan 'webpage' , there are :
>>> hbase(main):028:0> scan '1_webpage'
>>> ROW COLUMN+CELL
>>> com.xinhuanet.www:http/ column=f:fi, timestamp=1371110099941,
>>> value=\x00'\x8D\x00
>>> com.xinhuanet.www:http/ column=f:ts, timestamp=1371110099941,
>>> value=\x00\x00\x01?<\x87\xBA\x0A
>>> com.xinhuanet.www:http/ column=mk:_injmrk_,
>>> timestamp=1371110099941, value=y
>>> com.xinhuanet.www:http/ column=mk:dist,
>>> timestamp=1371110099941, value=0
>>> com.xinhuanet.www:http/ column=mtdt:_csh_,
>>> timestamp=1371110099941, value=?\x80\x00\x00
>>> com.xinhuanet.www:http/ column=s:s, timestamp=1371110099941,
>>> value=?\x80\x00\x00
>>> 1 row(s) in 0.0300 seconds
>>>
>>>
>>> So, is only 6 column are setted in the hbase ? And what is the real data
>>> stored in it?
>>> I find that in the source code, there is a WebPage Class. I could not
>>> understand all, but I think there should be 24 fileds in the hbase for each
>>> webside.
>>> public static final String[] _ALL_FIELDS =
>>> {"baseUrl","status","fetchTime","prevFetchTime","fetchInterval","retriesSinceFetch","modifiedTime","prevModifiedTime","protocolStatus","content","contentType","prevSignature","signature","title","text","parseStatus","score","reprUrl","headers","outlinks","inlinks","markers","metadata","batchId",};
>>>
>>>
>>> Thanks
>>> HeChuan
>>>
>>>
>
Re: what is stored in the hbase after inject job
Posted by Ahmet Emre Aladağ <em...@agmlab.com>.
Hi Tejas,
What about the scores? Where are they stored? I could not find.
On 06/13/2013 09:00 PM, Tejas Patil wrote:
> row-key = url
>
> column=f:fi : fetchInterval (the delay between re-fetches of a
> page)
> column=f:ts : fetchTime (indicates when the url will be elligible
> for fetching)
> column=mk:_injmrk_ : markers
> column=mk:dist
> column=mtdt:_csh_ : metadata
> column=s:s : status (is the url fetched, unfetched, newly
> injected, gone, redirected etc..)
>
>
>
> On Thu, Jun 13, 2013 at 6:40 AM, RS <ti...@163.com> wrote:
>
>> I do not what is sotred in the hbase after inject a website.
>> When I use the hbase shell $ scan 'webpage' , there are :
>> hbase(main):028:0> scan '1_webpage'
>> ROW COLUMN+CELL
>> com.xinhuanet.www:http/ column=f:fi, timestamp=1371110099941,
>> value=\x00'\x8D\x00
>> com.xinhuanet.www:http/ column=f:ts, timestamp=1371110099941,
>> value=\x00\x00\x01?<\x87\xBA\x0A
>> com.xinhuanet.www:http/ column=mk:_injmrk_,
>> timestamp=1371110099941, value=y
>> com.xinhuanet.www:http/ column=mk:dist,
>> timestamp=1371110099941, value=0
>> com.xinhuanet.www:http/ column=mtdt:_csh_,
>> timestamp=1371110099941, value=?\x80\x00\x00
>> com.xinhuanet.www:http/ column=s:s, timestamp=1371110099941,
>> value=?\x80\x00\x00
>> 1 row(s) in 0.0300 seconds
>>
>>
>> So, is only 6 column are setted in the hbase ? And what is the real data
>> stored in it?
>> I find that in the source code, there is a WebPage Class. I could not
>> understand all, but I think there should be 24 fileds in the hbase for each
>> webside.
>> public static final String[] _ALL_FIELDS =
>> {"baseUrl","status","fetchTime","prevFetchTime","fetchInterval","retriesSinceFetch","modifiedTime","prevModifiedTime","protocolStatus","content","contentType","prevSignature","signature","title","text","parseStatus","score","reprUrl","headers","outlinks","inlinks","markers","metadata","batchId",};
>>
>>
>> Thanks
>> HeChuan
>>
>>
Re: what is stored in the hbase after inject job
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,
On Thursday, June 13, 2013, RS <ti...@163.com> wrote:
> Thanks a lot
> 1.Is there a document discribe the column symbols (likecolumn=s:s )?
There are a lot symbols I can not understand.
Please check your gora-hbase-mapping.xml, this is where field names and
qualifiers are defined. It will be very valuable to you.
> 2.I find a problem in my new nutch project, when only run the
generate job in eclipse after inject job, there is no more
"column=mk:_gnmrk_" in the hbase, but when I use the nutch2.1, after
generate job, there will put this column in. Is this changed?
Can you please define your new Nutch project? Are you using 2.2, 2.x HEAD?
Thanks
Lewis
>
> Thanks
> HeChuan
>
>
> At 2013-06-14 02:00:24,"Tejas Patil" <te...@gmail.com> wrote:
>>row-key = url
>>
>>column=f:fi : fetchInterval (the delay between re-fetches of a
>>page)
>>column=f:ts : fetchTime (indicates when the url will be elligible
>>for fetching)
>>column=mk:_injmrk_ : markers
>>column=mk:dist
>>column=mtdt:_csh_ : metadata
>>column=s:s : status (is the url fetched, unfetched, newly
>>injected, gone, redirected etc..)
>>
>>
>>
>>On Thu, Jun 13, 2013 at 6:40 AM, RS <ti...@163.com> wrote:
>>
>>> I do not what is sotred in the hbase after inject a website.
>>> When I use the hbase shell $ scan 'webpage' , there are :
>>> hbase(main):028:0> scan '1_webpage'
>>> ROW COLUMN+CELL
>>> com.xinhuanet.www:http/ column=f:fi,
timestamp=1371110099941,
>>> value=\x00'\x8D\x00
>>> com.xinhuanet.www:http/ column=f:ts,
timestamp=1371110099941,
>>> value=\x00\x00\x01?<\x87\xBA\x0A
>>> com.xinhuanet.www:http/ column=mk:_injmrk_,
>>> timestamp=1371110099941, value=y
>>> com.xinhuanet.www:http/ column=mk:dist,
>>> timestamp=1371110099941, value=0
>>> com.xinhuanet.www:http/ column=mtdt:_csh_,
>>> timestamp=1371110099941, value=?\x80\x00\x00
>>> com.xinhuanet.www:http/ column=s:s,
timestamp=1371110099941,
>>> value=?\x80\x00\x00
>>> 1 row(s) in 0.0300 seconds
>>>
>>>
>>> So, is only 6 column are setted in the hbase ? And what is the real data
>>> stored in it?
>>> I find that in the source code, there is a WebPage Class. I could not
>>> understand all, but I think there should be 24 fileds in the hbase for
each
>>> webside.
>>> public static final String[] _ALL_FIELDS =
>>>
{"baseUrl","status","fetchTime","prevFetchTime","fetchInterval","retriesSinceFetch","modifiedTime","prevModifiedTime","protocolStatus","content","contentType","prevSignature","signature","title","text","parseStatus","score","reprUrl","headers","outlinks","inlinks","markers","metadata","batchId",};
>>>
>>>
>>> Thanks
>>> HeChuan
>>>
>>>
>
--
*Lewis*
Re:Re: what is stored in the hbase after inject job
Posted by RS <ti...@163.com>.
Thanks a lot
1.Is there a document discribe the column symbols (likecolumn=s:s )? There are a lot symbols I can not understand.
2.I find a problem in my new nutch project, when only run the generate job in eclipse after inject job, there is no more "column=mk:_gnmrk_" in the hbase, but when I use the nutch2.1, after generate job, there will put this column in. Is this changed?
Thanks
HeChuan
At 2013-06-14 02:00:24,"Tejas Patil" <te...@gmail.com> wrote:
>row-key = url
>
>column=f:fi : fetchInterval (the delay between re-fetches of a
>page)
>column=f:ts : fetchTime (indicates when the url will be elligible
>for fetching)
>column=mk:_injmrk_ : markers
>column=mk:dist
>column=mtdt:_csh_ : metadata
>column=s:s : status (is the url fetched, unfetched, newly
>injected, gone, redirected etc..)
>
>
>
>On Thu, Jun 13, 2013 at 6:40 AM, RS <ti...@163.com> wrote:
>
>> I do not what is sotred in the hbase after inject a website.
>> When I use the hbase shell $ scan 'webpage' , there are :
>> hbase(main):028:0> scan '1_webpage'
>> ROW COLUMN+CELL
>> com.xinhuanet.www:http/ column=f:fi, timestamp=1371110099941,
>> value=\x00'\x8D\x00
>> com.xinhuanet.www:http/ column=f:ts, timestamp=1371110099941,
>> value=\x00\x00\x01?<\x87\xBA\x0A
>> com.xinhuanet.www:http/ column=mk:_injmrk_,
>> timestamp=1371110099941, value=y
>> com.xinhuanet.www:http/ column=mk:dist,
>> timestamp=1371110099941, value=0
>> com.xinhuanet.www:http/ column=mtdt:_csh_,
>> timestamp=1371110099941, value=?\x80\x00\x00
>> com.xinhuanet.www:http/ column=s:s, timestamp=1371110099941,
>> value=?\x80\x00\x00
>> 1 row(s) in 0.0300 seconds
>>
>>
>> So, is only 6 column are setted in the hbase ? And what is the real data
>> stored in it?
>> I find that in the source code, there is a WebPage Class. I could not
>> understand all, but I think there should be 24 fileds in the hbase for each
>> webside.
>> public static final String[] _ALL_FIELDS =
>> {"baseUrl","status","fetchTime","prevFetchTime","fetchInterval","retriesSinceFetch","modifiedTime","prevModifiedTime","protocolStatus","content","contentType","prevSignature","signature","title","text","parseStatus","score","reprUrl","headers","outlinks","inlinks","markers","metadata","batchId",};
>>
>>
>> Thanks
>> HeChuan
>>
>>
Re: what is stored in the hbase after inject job
Posted by Tejas Patil <te...@gmail.com>.
row-key = url
column=f:fi : fetchInterval (the delay between re-fetches of a
page)
column=f:ts : fetchTime (indicates when the url will be elligible
for fetching)
column=mk:_injmrk_ : markers
column=mk:dist
column=mtdt:_csh_ : metadata
column=s:s : status (is the url fetched, unfetched, newly
injected, gone, redirected etc..)
On Thu, Jun 13, 2013 at 6:40 AM, RS <ti...@163.com> wrote:
> I do not what is sotred in the hbase after inject a website.
> When I use the hbase shell $ scan 'webpage' , there are :
> hbase(main):028:0> scan '1_webpage'
> ROW COLUMN+CELL
> com.xinhuanet.www:http/ column=f:fi, timestamp=1371110099941,
> value=\x00'\x8D\x00
> com.xinhuanet.www:http/ column=f:ts, timestamp=1371110099941,
> value=\x00\x00\x01?<\x87\xBA\x0A
> com.xinhuanet.www:http/ column=mk:_injmrk_,
> timestamp=1371110099941, value=y
> com.xinhuanet.www:http/ column=mk:dist,
> timestamp=1371110099941, value=0
> com.xinhuanet.www:http/ column=mtdt:_csh_,
> timestamp=1371110099941, value=?\x80\x00\x00
> com.xinhuanet.www:http/ column=s:s, timestamp=1371110099941,
> value=?\x80\x00\x00
> 1 row(s) in 0.0300 seconds
>
>
> So, is only 6 column are setted in the hbase ? And what is the real data
> stored in it?
> I find that in the source code, there is a WebPage Class. I could not
> understand all, but I think there should be 24 fileds in the hbase for each
> webside.
> public static final String[] _ALL_FIELDS =
> {"baseUrl","status","fetchTime","prevFetchTime","fetchInterval","retriesSinceFetch","modifiedTime","prevModifiedTime","protocolStatus","content","contentType","prevSignature","signature","title","text","parseStatus","score","reprUrl","headers","outlinks","inlinks","markers","metadata","batchId",};
>
>
> Thanks
> HeChuan
>
>