You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Vijith <vi...@gmail.com> on 2012/09/03 07:04:45 UTC
Running Junit test
Hi all,
How to run the junit test in nutch 1.5. I am trying to run the TestFetcher
from runtime/local and I am getting a 'no agents listed in http.agent.name'
error.
Which nutch-site.xml file is it reading. I dont have a config file in
local/test folder.
I have followed in instructions in
http://wiki.apache.org/nutch/bin/nutch%20junit
*
*
*Vijith V.*
Re: Running Junit test
Posted by feng lu <am...@gmail.com>.
Hi Vijith
in TestFetcher.java:136, the code like this
//verify that enough pages were handled
assertEquals(urls.size(), handledurls.size());
> AssertionFailedError:
> expected:<6> but was:<0>
here 6 indicate that there are six pages to generate and start to fetch,
and here 0 indicate that there are none page to fetch success. Maybe check
you fetch configuration or debug it to see what happen when fetch generate
pages.
On Wed, Sep 5, 2012 at 12:55 PM, Vijith <vi...@gmail.com> wrote:
> >
> > If you look at lines 395-399 in build.xml [0] you need to add
> >
> > <copy file="${test.src.dir}/crawl-tests.xml"
> > todir="${test.build.classes}"/>
> >
> > <copy file="${test.src.dir}/domain-urlfilter.txt"
> > todir="${test.build.classes}"/>
> >
> > <copy file="${test.src.dir}/filter-all.txt"
> > todir="${test.build.classes}"/>
> >
> > this will copy the desired config to the test directory.
> >
> > Again you need to do ant test *BEFORE* doing ant runtime
> >
> >
> Thanks again Lewis. Now it copied the files and the error has gone. But the
> test failed with 1 failure. This was for release 1.5 without any
> modification.
> ....
> ...
> >> Fetcher: finished at 2012-09-05 10:14:26, elapsed: 00:00:10
> >> F.Fetcher: No agents listed in 'http.agent.name' property.
>
> >> Time: 43.889
>
> >> There was 1 failure:
> >> 1)
>
> testFetch(org.apache.nutch.fetcher.TestFetcher)junit.framework.AssertionFailedError:
> expected:<6> but was:<0>
> >> at org.apache.nutch.fetcher.TestFetcher.testFetch(TestFetcher.java:136)
> ...
>
> why was that ? what does handleUrls store ?
>
> Are you able to submit a patch for this?
> >
> >
> you mean a patch for the build.xml file... surely I can.
>
>
> > Thanks
> >
> > Lewis
> >
> > [0] http://svn.apache.org/viewvc/nutch/trunk/build.xml?view=markup
> >
> > On Tue, Sep 4, 2012 at 1:31 PM, Vijith <vi...@gmail.com> wrote:
> > > Thanks Lewis. But even then I am getting the same error. Here is what
> my
> > > hadoop.log shows
> > > ...
> >
>
>
>
> --
>
> *Vijith V.*
>
--
Don't Grow Old, Grow Up... :-)
Re: Running Junit test
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Vijith,
On Wed, Sep 5, 2012 at 5:55 AM, Vijith <vi...@gmail.com> wrote:
>
> Are you able to submit a patch for this?
>>
>>
> you mean a patch for the build.xml file... surely I can.
>
Excellent :0)
I also noticed that there is currently no way to copy the compiled
plugin test cases through to the runtime directory therefore in order
to test plugins we currently need to test ALL of them, which again is
a pain.
Lewis
Re: Running Junit test
Posted by Vijith <vi...@gmail.com>.
>
> If you look at lines 395-399 in build.xml [0] you need to add
>
> <copy file="${test.src.dir}/crawl-tests.xml"
> todir="${test.build.classes}"/>
>
> <copy file="${test.src.dir}/domain-urlfilter.txt"
> todir="${test.build.classes}"/>
>
> <copy file="${test.src.dir}/filter-all.txt"
> todir="${test.build.classes}"/>
>
> this will copy the desired config to the test directory.
>
> Again you need to do ant test *BEFORE* doing ant runtime
>
>
Thanks again Lewis. Now it copied the files and the error has gone. But the
test failed with 1 failure. This was for release 1.5 without any
modification.
....
...
>> Fetcher: finished at 2012-09-05 10:14:26, elapsed: 00:00:10
>> F.Fetcher: No agents listed in 'http.agent.name' property.
>> Time: 43.889
>> There was 1 failure:
>> 1)
testFetch(org.apache.nutch.fetcher.TestFetcher)junit.framework.AssertionFailedError:
expected:<6> but was:<0>
>> at org.apache.nutch.fetcher.TestFetcher.testFetch(TestFetcher.java:136)
...
why was that ? what does handleUrls store ?
Are you able to submit a patch for this?
>
>
you mean a patch for the build.xml file... surely I can.
> Thanks
>
> Lewis
>
> [0] http://svn.apache.org/viewvc/nutch/trunk/build.xml?view=markup
>
> On Tue, Sep 4, 2012 at 1:31 PM, Vijith <vi...@gmail.com> wrote:
> > Thanks Lewis. But even then I am getting the same error. Here is what my
> > hadoop.log shows
> > ...
>
--
*Vijith V.*
Re: Running Junit test
Posted by Lewis John Mcgibbney <le...@gmail.com>.
If you look at lines 395-399 in build.xml [0] you need to add
<copy file="${test.src.dir}/crawl-tests.xml"
todir="${test.build.classes}"/>
<copy file="${test.src.dir}/domain-urlfilter.txt"
todir="${test.build.classes}"/>
<copy file="${test.src.dir}/filter-all.txt"
todir="${test.build.classes}"/>
this will copy the desired config to the test directory.
Again you need to do ant test *BEFORE* doing ant runtime
Are you able to submit a patch for this?
Thanks
Lewis
[0] http://svn.apache.org/viewvc/nutch/trunk/build.xml?view=markup
On Tue, Sep 4, 2012 at 1:31 PM, Vijith <vi...@gmail.com> wrote:
> Thanks Lewis. But even then I am getting the same error. Here is what my
> hadoop.log shows
> ...
Re: Running Junit test
Posted by Vijith <vi...@gmail.com>.
Thanks Lewis. But even then I am getting the same error. Here is what my
hadoop.log shows
...
2012-09-04 17:17:50,246 INFO crawl.Generator - Generator: finished at
2012-09-04 17:17:50, elapsed: 00:00:15
2012-09-04 17:17:50,252 ERROR fetcher.Fetcher - Fetcher: No agents listed
in 'http.agent.name' property.
2012-09-04 17:17:50,253 INFO mortbay.log - Stopped
SocketConnector@127.0.0.1:50000
2012-09-04 17:17:50,300 INFO mortbay.log - jetty-6.1.26
2012-09-04 17:17:50,304 INFO mortbay.log - Started
SocketConnector@127.0.0.1:50000
2012-09-04 17:17:50,304 ERROR fetcher.Fetcher - Fetcher: No agents listed
in 'http.agent.name' property.
2012-09-04 17:17:50,305 INFO mortbay.log - Stopped
SocketConnector@127.0.0.1:50000
2012-09-04 17:32:31,068 INFO mortbay.log - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-09-04 17:32:31,182 INFO mortbay.log - jetty-6.1.26
2012-09-04 17:32:31,223 INFO mortbay.log - Started
SocketConnector@127.0.0.1:50000
I am having the agent name and robot.agents set in
runtime/local/test/nutch-site.xml.
On Mon, Sep 3, 2012 at 2:44 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Before doing runtime local you need to ensure the test are executed
> and all of the resources are present in the build directory.
>
> So please do ant test, then ant runtime, all of the test resources
> should then be moved to the runtime/local directory.
>
> The runtime target does NOT rely on the test target because tests are
> expensive to run.
>
> hth
>
> Lewis
>
> On Mon, Sep 3, 2012 at 6:04 AM, Vijith <vi...@gmail.com> wrote:
> > Hi all,
> >
> > How to run the junit test in nutch 1.5. I am trying to run the
> TestFetcher
> > from runtime/local and I am getting a 'no agents listed in
> http.agent.name'
> > error.
> > Which nutch-site.xml file is it reading. I dont have a config file in
> > local/test folder.
> > I have followed in instructions in
> > http://wiki.apache.org/nutch/bin/nutch%20junit
> >
> >
> > *
> > *
> > *Vijith V.*
>
>
>
> --
> Lewis
>
--
*. . . . . thanks & regards*
*
*
*Vijith V.*
Re: Running Junit test
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Before doing runtime local you need to ensure the test are executed
and all of the resources are present in the build directory.
So please do ant test, then ant runtime, all of the test resources
should then be moved to the runtime/local directory.
The runtime target does NOT rely on the test target because tests are
expensive to run.
hth
Lewis
On Mon, Sep 3, 2012 at 6:04 AM, Vijith <vi...@gmail.com> wrote:
> Hi all,
>
> How to run the junit test in nutch 1.5. I am trying to run the TestFetcher
> from runtime/local and I am getting a 'no agents listed in http.agent.name'
> error.
> Which nutch-site.xml file is it reading. I dont have a config file in
> local/test folder.
> I have followed in instructions in
> http://wiki.apache.org/nutch/bin/nutch%20junit
>
>
> *
> *
> *Vijith V.*
--
Lewis