You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Law-Firms-In.com" <we...@law-firms-in.com> on 2013/09/26 20:38:35 UTC
Boilerpipe (NUCTH-961) with Nutch 2.2.1
My setup is:
*Nutch 2.2.1**
**Solr 4.4**
**Hbase 0.90.6*
I try to activate the boilerpipe support & I have done the following:
1) https://issues.apache.org/jira/browse/NUTCH-961
Applied the patch and edited manually TikaParser.java where the patch
did not work. Now when I type
"ant runtime" and try to compile nutch again I get the following error:
[javac]
/srv/apache-nutch-2.2.1/src/plugin/parse-tika/src/java/org/apache/nutch/parse/tika/TikaParser.java:176:
error: no suitable constructor found for ByteArrayInputStream(ByteBuffer)
[javac] parser.parse(new ByteArrayInputStream(raw),
(ContentHandler)domHandler, tikamd, context);
[javac] ^
[javac] constructor
ByteArrayInputStream.ByteArrayInputStream(byte[],int,int) is not applicable
[javac] (actual and formal argument lists differ in length)
[javac] constructor
ByteArrayInputStream.ByteArrayInputStream(byte[]) is not applicable
[javac] (actual argument ByteBuffer cannot be converted to
byte[] by method invocation conversion)
Now I have looked into the TikaParse.java file once again and looked at
the line which gives the error:
parser.parse(new ByteArrayInputStream*(raw)*, domHandler, tikamd,
context);
and I have tried to replace (raw) with several options, eg:
*raw.array(), raw.arrayOffset() + raw.position(), raw.remaining()*
but still no luck. _Although it compiles fine with the edit above_ when
I run Nutch _I dont get my content boilerpiped_ :(
I also have changed the *parse-plugins.xml* like this:
*<mimeType name="*"> **
**<plugin id="parse-tika" /> **
**</mimeType> **
**<mimeType name="text/html"> **
**<plugin id="parse-tika" /> **
**</mimeType> **
**<mimeType name="application/xhtml+xml"> **
**<plugin id="parse-tika" /> **
**</mimeType>**
*
and edited *nutch-site.xml*:
*<property> **
**<name>tika.use_boilerpipe</name> **
**<value>true</value> </property> **
**<property> **
**<name>tika.boilerpipe.extractor</name> **
**<value>ArticleExtractor</value> **
**</property>
*Any help or advise would be welcomed. I am since 10+ hours working on
it and have studied any tutorial, hint on the mailinglist/web before. I
am quite sure the problem is within
TikaParser.java -> Line: parser.parse(new ByteArrayInputStream*(raw)*,
domHandler, tikamd, context);
Thank you!
Re: AdaptiveFetchSchedule Test case
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi Yasin,
> * where is Webpage modifiedDate set? and how is it set ?
It's definitely set by one of {Adaptive,Default}FetchSchedule,
called by DbUpdateReducer.reduce().
> * can anyone please tell me about sync_delta? how does sync_delta work?
It is best explained in AdaptiveFetchSchedule.java itself.
But be aware: it's buggy, see NUTCH-1564. It would be nice to
see the bug reproducible in a unit test. Validation of patch
then comes for free.
There is also NUTCH-1502 which will also cover fetch schedulers
to some extend. But it's on a higher, complementary level,
testing update reduce function including scheduler implementations.
I hope to get a working patch ready soon, prelimary tests can be
found in NUTCH-1245.
Sebastian
On 09/30/2013 10:13 AM, Yasin Kılınç wrote:
> Ok. Feng,
>
> Before write to Test for Adaptive Fetch Schedule,
>
> * where is Webpage modifiedDate set? and how is it set ?
> * can anyone please tell me about sync_delta? how does sync_delta work?
>
>
> Thanks.
>
>
>
> On 29-09-2013 16:56, feng lu wrote:
>> On Fri, Sep 27, 2013 at 8:36 PM, Yasin Kılınç <ya...@agmlab.com>wrote:
>>
>>> Adaptive Fetch Schedule
>>
>> Hi Yasin
>>
>> currently there are no Test Case for Adaptive Fetch Schedule class, I only
>> find one main method in AdaptiveFetchSchedule class at Nutch 1.x.
>>
>> very welcome to write test case for AdaptiveFetchSchedule if you want to
>> and I will very happy to do check for you. you can open an issue in Jira.
>>
>>
>>
>>
>
Re: AdaptiveFetchSchedule Test case
Posted by Yasin Kılınç <ya...@agmlab.com>.
Ok. Feng,
Before write to Test for Adaptive Fetch Schedule,
* where is Webpage modifiedDate set? and how is it set ?
* can anyone please tell me about sync_delta? how does sync_delta work?
Thanks.
On 29-09-2013 16:56, feng lu wrote:
> On Fri, Sep 27, 2013 at 8:36 PM, Yasin Kılınç <ya...@agmlab.com>wrote:
>
>> Adaptive Fetch Schedule
>
> Hi Yasin
>
> currently there are no Test Case for Adaptive Fetch Schedule class, I only
> find one main method in AdaptiveFetchSchedule class at Nutch 1.x.
>
> very welcome to write test case for AdaptiveFetchSchedule if you want to
> and I will very happy to do check for you. you can open an issue in Jira.
>
>
>
>
Re: AdaptiveFetchSchedule Test case
Posted by feng lu <am...@gmail.com>.
On Fri, Sep 27, 2013 at 8:36 PM, Yasin Kılınç <ya...@agmlab.com>wrote:
> Adaptive Fetch Schedule
Hi Yasin
currently there are no Test Case for Adaptive Fetch Schedule class, I only
find one main method in AdaptiveFetchSchedule class at Nutch 1.x.
very welcome to write test case for AdaptiveFetchSchedule if you want to
and I will very happy to do check for you. you can open an issue in Jira.
--
Don't Grow Old, Grow Up... :-)
AdaptiveFetchSchedule Test case
Posted by Yasin Kılınç <ya...@agmlab.com>.
hi,
I want to use Adaptive Fetch Schedule in Nutch 2.2.1.
Is there any test case for this class? how can I test this class?
any idea?