You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Arthur Chan <ar...@gmail.com> on 2015/04/24 16:06:00 UTC
Nutch 2.3.1 HBASE Invalid Field Values
Hi,
My Nutch is 2.3 with Gora and Hbase, below are the sample field values I
have scanned from HBase here:
baseUrl: value=http://www.apache.org
status: value=\x00\x00\x00\x01
prevFetchTime: value=\x00\x00\x01L\x91]\xF5\x1C
fetchTime: value=\x00\x00\x01L\x93\x92\x0F\x5C
fetchInterval: value=\x00'\x8D\x00
retriesSinceFetch: \x00\x00\x00\x00
reprUrl: value=http://www.apache.org
protocolStatus: value=\x02\x00\x00
modifiedTime: value=\x00\x00\x01L\x93@\xE1H
prevModifiedTime: value=\x00\x00\x00\x00\x00\x00\x00\x00
batchId: value=1428399528-1598360492
parseStatus: value=\x02\x00\x00
signature: value=\xD7\xA7\x04pT7?E\xFA\x1A\x01"\x08\x89$0
prevSignature: value=\x85\xC2i@\xFC(\xDE\xEEt?\xE7\xFB\xE1rY\xAF
score: value=\x00\x00\x00\x00
Below is my related gora-hbase-mapping.xml about these fields
<field name="baseUrl" family="f" qualifier="bas"/>
<field name="status" family="f" qualifier="st"/>
<field name="prevFetchTime" family="f" qualifier="pts"/>
<field name="fetchTime" family="f" qualifier="ts"/>
<field name="fetchInterval" family="f" qualifier="fi"/>
<field name="retriesSinceFetch" family="f" qualifier="rsf"/>
<field name="reprUrl" family="f" qualifier="rpr"/>
<field name="content" family="f" qualifier="cnt"/>
<field name="contentType" family="f" qualifier="typ"/>
<field name="protocolStatus" family="f" qualifier="prot"/>
<field name="modifiedTime" family="f" qualifier="mod"/>
<field name="prevModifiedTime" family="f" qualifier="pmod"/>
<field name="batchId" family="f" qualifier="bid"/>
<field name="title" family="p" qualifier="t"/>
<field name="text" family="p" qualifier="c"/>
<field name="parseStatus" family="p" qualifier="st"/>
<field name="signature" family="p" qualifier="sig"/>
<field name="prevSignature" family="p" qualifier="psig"/>
<field name="score" family="s" qualifier="s"/>
Q: Is there a way to configure Nutch/Gora/HBase so it will store the value
like following and no need to do field type conversion?
baseUrl: null
status: 4 (status_redir_temp)
fetchTime: 1426888912463
prevFetchTime: 1424296904936
fetchInterval: 2592000
retriesSinceFetch: 0
modifiedTime: 0
prevModifiedTime: 0
protocolStatus: (null)
parseStatus: (null)
title: null
score: 1.0
marker _injmrk_ : y
marker dist : 0
reprUrl: null
batchId: 1424296906-20007
Please help!
Regards
Re: Nutch 2.3.1 HBASE Invalid Field Values
Posted by Arthur Chan <ar...@gmail.com>.
Is my configuration incorrect? any idea why the value, like fetchTime,
looks strange here?
On Fri, Apr 24, 2015 at 10:06 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> My Nutch is 2.3 with Gora and Hbase, below are the sample field values I
> have scanned from HBase here:
>
> baseUrl: value=http://www.apache.org
> status: value=\x00\x00\x00\x01
> prevFetchTime: value=\x00\x00\x01L\x91]\xF5\x1C
> fetchTime: value=\x00\x00\x01L\x93\x92\x0F\x5C
> fetchInterval: value=\x00'\x8D\x00
> retriesSinceFetch: \x00\x00\x00\x00
> reprUrl: value=http://www.apache.org
> protocolStatus: value=\x02\x00\x00
> modifiedTime: value=\x00\x00\x01L\x93@\xE1H
> prevModifiedTime: value=\x00\x00\x00\x00\x00\x00\x00\x00
> batchId: value=1428399528-1598360492
> parseStatus: value=\x02\x00\x00
> signature: value=\xD7\xA7\x04pT7?E\xFA\x1A\x01"\x08\x89$0
> prevSignature: value=\x85\xC2i@\xFC(\xDE\xEEt?\xE7\xFB\xE1rY\xAF
> score: value=\x00\x00\x00\x00
>
>
>
>
> Below is my related gora-hbase-mapping.xml about these fields
>
> <field name="baseUrl" family="f" qualifier="bas"/>
> <field name="status" family="f" qualifier="st"/>
> <field name="prevFetchTime" family="f" qualifier="pts"/>
> <field name="fetchTime" family="f" qualifier="ts"/>
> <field name="fetchInterval" family="f" qualifier="fi"/>
> <field name="retriesSinceFetch" family="f" qualifier="rsf"/>
> <field name="reprUrl" family="f" qualifier="rpr"/>
> <field name="content" family="f" qualifier="cnt"/>
> <field name="contentType" family="f" qualifier="typ"/>
> <field name="protocolStatus" family="f" qualifier="prot"/>
> <field name="modifiedTime" family="f" qualifier="mod"/>
> <field name="prevModifiedTime" family="f" qualifier="pmod"/>
> <field name="batchId" family="f" qualifier="bid"/>
> <field name="title" family="p" qualifier="t"/>
> <field name="text" family="p" qualifier="c"/>
> <field name="parseStatus" family="p" qualifier="st"/>
> <field name="signature" family="p" qualifier="sig"/>
> <field name="prevSignature" family="p" qualifier="psig"/>
> <field name="score" family="s" qualifier="s"/>
>
>
>
> Q: Is there a way to configure Nutch/Gora/HBase so it will store the value
> like following and no need to do field type conversion?
>
> baseUrl: null
> status: 4 (status_redir_temp)
> fetchTime: 1426888912463
> prevFetchTime: 1424296904936
> fetchInterval: 2592000
> retriesSinceFetch: 0
> modifiedTime: 0
> prevModifiedTime: 0
> protocolStatus: (null)
> parseStatus: (null)
> title: null
> score: 1.0
> marker _injmrk_ : y
> marker dist : 0
> reprUrl: null
> batchId: 1424296906-20007
>
>
> Please help!
>
> Regards
>
>