You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by EM <em...@cpuedge.com> on 2005/08/10 00:17:57 UTC
strange url counting in the fetcher

Should the following be happening?

Short description:
-fetch bunch of pages.
-status: 5400 fetched, 27 errors
-fetch 22 more pages
-status: 5403 fetched, 27 errors

My regex-urlfilter excludes "jpg" and includes "?"

Long description:

050809 093001 fetching http://<domain>/<text>.jpg?6351
050809 093001 fetching http://<domain>/<text>.jpg?4141
050809 093001 fetching http://<domain>/<text>.jpg?4333
050809 093001 fetching http://<domain>/<text>.htm
050809 093002 status: segment 20050808224036-5, 5400 pages, 27 errors,
40986108 bytes, 30626719 ms
050809 093002 status: 0.17631662 pages/s, 10.455052 kb/s, 7590.02 bytes/page
050809 093002 fetching http://<domain>/<text>.jpg?3732
050809 093002 fetching http://<domain>/<text>.jpg?1398
050809 093002 fetching http://<domain>/<text>.jpg?3876
050809 093002 fetching http://<domain>/<text>.jpg?2260
050809 093002 fetching http://<domain>/<text>.jpg?3298
050809 093002 fetching http://<domain>/<text>.jpg?9396
050809 093002 fetching http://<domain>/<text>.jpg?1946
050809 093002 fetching http://<domain>/<text>.jpg?9897
050809 093002 fetching http://<domain>/<text>>.htm
050809 093007 Response content length is not known
050809 093007 fetching http:// <domain>/<text>.htm
050809 093014 Response content length is not known
050809 093015 fetching http://<domain>/<text>.jpg?8507
050809 093015 fetching http:// <domain>/<text>.htm
050809 093022 Response content length is not known
050809 093023 fetching http://<domain>/<text>.jpg?693
050809 093023 fetching http://<domain>/<text>.jpg?4637
050809 093023 fetching http://<domain>/<text>.jpg?7929
050809 093023 fetching http://<domain>/<text>.jpg?7113
050809 093023 fetching http://<domain>/<text>.jpg?6956
050809 093023 fetching http://<domain>/<text>.jpg?5054
050809 093023 fetching http://<domain>/<text>.jpg?4768
050809 093023 fetching http://<domain>/<text>.jpg?1673
050809 093023 fetching http://<domain>/<text>.jpg?2583
050809 093023 fetching http://<domain>/<text>.jpg?4526
050809 093024 status: segment 20050808224036-5, 5403 pages, 27 errors,
41002564 bytes, 30648859 ms
050809 093024 status: 0.17628714 pages/s, 10.451695 kb/s, 7588.8516
bytes/page