You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Law-Firms-In.com" <we...@law-firms-in.com> on 2013/09/26 20:38:35 UTC

Boilerpipe (NUCTH-961) with Nutch 2.2.1

My setup is:

*Nutch 2.2.1**
**Solr 4.4**
**Hbase 0.90.6*

I try to activate the boilerpipe support & I have done the following:

1) https://issues.apache.org/jira/browse/NUTCH-961

Applied the patch and edited manually TikaParser.java where the patch
did not work. Now when I type
"ant runtime" and try to compile nutch again I get the following error:

    [javac]
/srv/apache-nutch-2.2.1/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/TikaParser.java:176:
error: no suitable constructor found for ByteArrayInputStream(ByteBuffer)
    [javac]     parser.parse(new ByteArrayInputStream(raw),
(ContentHandler)domHandler, tikamd, context);
    [javac]                  ^
    [javac]     constructor
ByteArrayInputStream.ByteArrayInputStream(byte[],int,int) is not applicable
    [javac]       (actual and formal argument lists differ in length)
    [javac]     constructor
ByteArrayInputStream.ByteArrayInputStream(byte[]) is not applicable
    [javac]       (actual argument ByteBuffer cannot be converted to
byte[] by method invocation conversion)

Now I have looked into the TikaParse.java file once again and looked at
the line which gives the error:

    parser.parse(new ByteArrayInputStream*(raw)*, domHandler, tikamd,
context);

and I have tried to replace (raw) with several options, eg: 
*raw.array(), raw.arrayOffset() + raw.position(), raw.remaining()*
   
but still no luck. _Although it compiles fine with the edit above_ when
I run Nutch _I dont get my content boilerpiped_ :(

I also have changed the *parse-plugins.xml* like this:

*<mimeType name="*"> **
**<plugin id="parse-tika" /> **
**</mimeType> **
**<mimeType name="text/html"> **
**<plugin id="parse-tika" /> **
**</mimeType> **
**<mimeType name="application/xhtml+xml"> **
**<plugin id="parse-tika" /> **
**</mimeType>**
*
and edited *nutch-site.xml*:

*<property> **
**<name>tika.use_boilerpipe</name> **
**<value>true</value> </property> **
**<property> **
**<name>tika.boilerpipe.extractor</name> **
**<value>ArticleExtractor</value> **
**</property>

*Any help or advise would be welcomed. I am since 10+ hours working on
it and have studied any tutorial, hint on the mailinglist/web before. I
am quite sure the problem is within
TikaParser.java -> Line: parser.parse(new ByteArrayInputStream*(raw)*,
domHandler, tikamd, context);

Thank you!














Re: AdaptiveFetchSchedule Test case

Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Yasin,

> * where is Webpage modifiedDate set? and how is it set ?
It's definitely set by one of {Adaptive,Default}FetchSchedule,
called by DbUpdateReducer.reduce().

> * can  anyone please tell me about sync_delta? how does sync_delta work?
It is best explained in AdaptiveFetchSchedule.java itself.
But be aware: it's buggy, see NUTCH-1564. It would be nice to
see the bug reproducible in a unit test. Validation of patch
then comes for free.

There is also NUTCH-1502 which will also cover fetch schedulers
to some extend. But it's on a higher, complementary level,
testing update reduce function including scheduler implementations.
I hope to get a working patch ready soon, prelimary tests can be
found in NUTCH-1245.

Sebastian

On 09/30/2013 10:13 AM, Yasin Kılınç wrote:
> Ok. Feng,
> 
> Before write to Test for Adaptive Fetch Schedule,
> 
> * where is Webpage modifiedDate set? and how is it set ?
> * can  anyone please tell me about sync_delta? how does sync_delta work?
> 
> 
> Thanks.
> 
> 
> 
> On 29-09-2013 16:56, feng lu wrote:
>> On Fri, Sep 27, 2013 at 8:36 PM, Yasin Kılınç <ya...@agmlab.com>wrote:
>>
>>> Adaptive Fetch Schedule
>>
>> Hi Yasin
>>
>> currently there are no Test Case for Adaptive Fetch Schedule class, I only
>> find one main method in AdaptiveFetchSchedule class at Nutch 1.x.
>>
>> very welcome to write test case for AdaptiveFetchSchedule if you want to
>> and I will very happy to do check for you. you can open an issue in Jira.
>>
>>
>>
>>
> 


Re: AdaptiveFetchSchedule Test case

Posted by Yasin Kılınç <ya...@agmlab.com>.
Ok. Feng,

Before write to Test for Adaptive Fetch Schedule,

* where is Webpage modifiedDate set? and how is it set ?
* can  anyone please tell me about sync_delta? how does sync_delta work?


Thanks.



On 29-09-2013 16:56, feng lu wrote:
> On Fri, Sep 27, 2013 at 8:36 PM, Yasin Kılınç <ya...@agmlab.com>wrote:
>
>> Adaptive Fetch Schedule
>
> Hi Yasin
>
> currently there are no Test Case for Adaptive Fetch Schedule class, I only
> find one main method in AdaptiveFetchSchedule class at Nutch 1.x.
>
> very welcome to write test case for AdaptiveFetchSchedule if you want to
> and I will very happy to do check for you. you can open an issue in Jira.
>
>
>
>


Re: AdaptiveFetchSchedule Test case

Posted by feng lu <am...@gmail.com>.
On Fri, Sep 27, 2013 at 8:36 PM, Yasin Kılınç <ya...@agmlab.com>wrote:

> Adaptive Fetch Schedule


Hi Yasin

currently there are no Test Case for Adaptive Fetch Schedule class, I only
find one main method in AdaptiveFetchSchedule class at Nutch 1.x.

very welcome to write test case for AdaptiveFetchSchedule if you want to
and I will very happy to do check for you. you can open an issue in Jira.




-- 
Don't Grow Old, Grow Up... :-)

AdaptiveFetchSchedule Test case

Posted by Yasin Kılınç <ya...@agmlab.com>.
hi,

I want to use Adaptive Fetch Schedule in Nutch 2.2.1.

Is there any test case for this class? how can I test this class?

any idea?