No parsed text with hbase backend & no solr doc.

[Tony Mullins <to...@gmail.com>]

Hi,

I have successfully setup nutch 2.x with hbase-0.90.6 and my jobs are
running fine. But there is one issue for which I need your help.

Earlier I was using Cassandra with nutch 2.x and data from my all jobs were
used to go to 'webpage'  keyspace. But in case of hbase-0.90.6 I can see
there are 2 tables created , one is 'webpage' which always have 0 rows and
other is 'crawlId_webpage' and that has some data.

But when I run my solrIndexJob ,  no documents are added and I think this
is due to the face that there is no parsed text present in
'crawlId_webpage' table for my crawled pages.

I can also verify this in my ParseFilter plugin when I do Utf8 text =
page.getText();
my text is always null and thats why I think solrindexjob is not inserting
any doc to Solr.

So what should I do here ? Why I am not having any text in hbase table ?
And why there are two tables created 'webpage' & 'crawlid_webpage' ?

Thanks guys for help & support.

Tony.

Re: No parsed text with hbase backend & no solr doc.

[Tony Mullins <to...@gmail.com>]

If I dont give any crawlId to my Nutch jobs then it creates and populate
'webpage' table and my solrIndexJob also execute successfully.

But what if I want to crawl specific site only and for that I need to give
a new crawlId to my Nutch job and when I will give a crawlId it will create
new 'crawlId_webpage' table and solrindexJob will be failed as it wont know
how to and from where get its documents.

So how do guyz do id based crawling with hbase ?

Thanks,
Tony





On Mon, Jun 24, 2013 at 4:36 PM, Tony Mullins <to...@gmail.com>wrote:

> Ok, I have crawled again . And on checking my hbase  "scan 'C23_webpage',
> {COLUMNS => ['p:c']}" I can see the parsed text but in my ParseFilter
> plugin I still don't get null for page.getText().
>
> But disturbing thing for me is that why there is blank 'webpage' table and
> for every new crawlId I get new 'crawld_webpage' table ?
> And my SolrIndex job was working fine with cassandra but its not working
> with hbase now , is this due to different table structure in hbase, how
> should I solve it ?
>
> Thanks,
> Tony
>
>
> On Mon, Jun 24, 2013 at 2:39 PM, Tony Mullins <to...@gmail.com>wrote:
>
>> Hi,
>>
>> I have successfully setup nutch 2.x with hbase-0.90.6 and my jobs are
>> running fine. But there is one issue for which I need your help.
>>
>> Earlier I was using Cassandra with nutch 2.x and data from my all jobs
>> were used to go to 'webpage'  keyspace. But in case of hbase-0.90.6 I can
>> see there are 2 tables created , one is 'webpage' which always have 0 rows
>> and other is 'crawlId_webpage' and that has some data.
>>
>> But when I run my solrIndexJob ,  no documents are added and I think this
>> is due to the face that there is no parsed text present in
>> 'crawlId_webpage' table for my crawled pages.
>>
>> I can also verify this in my ParseFilter plugin when I do Utf8 text =
>> page.getText();
>> my text is always null and thats why I think solrindexjob is not
>> inserting any doc to Solr.
>>
>> So what should I do here ? Why I am not having any text in hbase table ?
>> And why there are two tables created 'webpage' & 'crawlid_webpage' ?
>>
>> Thanks guys for help & support.
>>
>> Tony.
>>
>
>

Re: No parsed text with hbase backend & no solr doc.

[Tony Mullins <to...@gmail.com>]

Ok, I have crawled again . And on checking my hbase  "scan 'C23_webpage',
{COLUMNS => ['p:c']}" I can see the parsed text but in my ParseFilter
plugin I still don't get null for page.getText().

But disturbing thing for me is that why there is blank 'webpage' table and
for every new crawlId I get new 'crawld_webpage' table ?
And my SolrIndex job was working fine with cassandra but its not working
with hbase now , is this due to different table structure in hbase, how
should I solve it ?

Thanks,
Tony


On Mon, Jun 24, 2013 at 2:39 PM, Tony Mullins <to...@gmail.com>wrote:

> Hi,
>
> I have successfully setup nutch 2.x with hbase-0.90.6 and my jobs are
> running fine. But there is one issue for which I need your help.
>
> Earlier I was using Cassandra with nutch 2.x and data from my all jobs
> were used to go to 'webpage'  keyspace. But in case of hbase-0.90.6 I can
> see there are 2 tables created , one is 'webpage' which always have 0 rows
> and other is 'crawlId_webpage' and that has some data.
>
> But when I run my solrIndexJob ,  no documents are added and I think this
> is due to the face that there is no parsed text present in
> 'crawlId_webpage' table for my crawled pages.
>
> I can also verify this in my ParseFilter plugin when I do Utf8 text =
> page.getText();
> my text is always null and thats why I think solrindexjob is not inserting
> any doc to Solr.
>
> So what should I do here ? Why I am not having any text in hbase table ?
> And why there are two tables created 'webpage' & 'crawlid_webpage' ?
>
> Thanks guys for help & support.
>
> Tony.
>