You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Rozina Sorathia <Ro...@KPITCummins.com> on 2006/01/04 15:18:27 UTC
Problem crawling with nutch-08
Hi
I am trying to crawl using nutch-08 presently my urls.txt in the urls
folder has just one url as I am just testing it to know the structure
of segments and index with nutch-08 to be able to search using nutch-08
but everytime I do crawling it goes in a loop.It is not throwing any
exception but even for a single url the crawling is taking over 5 hours.
Is something going wrong....Also if some one who has successfully
crawled with nutch 08 and created segments and index can give some idea
how these actually are?
060104 193547 map -1057068992%
060104 193548 urls\urls.txt:0+94
060104 193548 map -1057126976%
060104 193549 urls\urls.txt:0+94
060104 193549 map -1057180928%
060104 193550 urls\urls.txt:0+94
060104 193550 map -1057240320%
060104 193551 urls\urls.txt:0+94
060104 193551 map -1057299392%
060104 193552 urls\urls.txt:0+94
060104 193552 map -1057359232%
060104 193553 urls\urls.txt:0+94
060104 193553 map -1057417280%
060104 193554 urls\urls.txt:0+94
060104 193554 map -1057474688%
060104 193555 urls\urls.txt:0+94
060104 193555 map -1057533888%
060104 193556 urls\urls.txt:0+94
060104 193556 map -1057588800%
060104 193557 urls\urls.txt:0+94
060104 193557 map -1057645824%
060104 193558 urls\urls.txt:0+94
060104 193558 map -1057703104%
060104 193559 urls\urls.txt:0+94
060104 193600 map -1057761920%
060104 193600 urls\urls.txt:0+94
060104 193601 map -1057813824%
060104 193601 urls\urls.txt:0+94
060104 193602 map -1057871488%
060104 193602 urls\urls.txt:0+94
060104 193603 map -1057929728%
060104 193603 urls\urls.txt:0+94
060104 193604 map -1057979328%
060104 193604 urls\urls.txt:0+94
060104 193605 map -1058035328%
060104 193605 urls\urls.txt:0+94
060104 193606 map -1058091520%
060104 193606 urls\urls.txt:0+94
060104 193607 map -1058128128%
060104 193607 urls\urls.txt:0+94
060104 193608 map -1058164096%
060104 193608 urls\urls.txt:0+94
060104 193609 map -1058200512%
060104 193609 urls\urls.txt:0+94
060104 193610 map -1058233024%
060104 193610 urls\urls.txt:0+94
060104 193611 map -1058272384%
060104 193611 urls\urls.txt:0+94
060104 193612 map -1058318592%
060104 193612 urls\urls.txt:0+94
060104 193613 map -1058371008%
060104 193613 urls\urls.txt:0+94
060104 193614 map -1058416000%
060104 193614 urls\urls.txt:0+94
060104 193615 map -1058475200%
060104 193615 urls\urls.txt:0+94
060104 193616 map -1058533504%
060104 193616 urls\urls.txt:0+94
060104 193617 map -1058593280%
060104 193617 urls\urls.txt:0+94
060104 193618 map -1058650816%
060104 193618 urls\urls.txt:0+94
060104 193619 map -1058709312%
060104 193619 urls\urls.txt:0+94
060104 193620 map -1058765184%
060104 193620 urls\urls.txt:0+94
060104 193621 map -1058820288%
060104 193621 urls\urls.txt:0+94
060104 193622 map -1058877184%
060104 193622 urls\urls.txt:0+94
060104 193623 map -1058929728%
060104 193623 urls\urls.txt:0+94
060104 193624 map -1058987904%
060104 193624 urls\urls.txt:0+94
060104 193625 map -1059045632%
060104 193625 urls\urls.txt:0+94
060104 193626 map -1059104000%
060104 193626 urls\urls.txt:0+94
060104 193627 map -1059163520%
060104 193627 urls\urls.txt:0+94
060104 193628 map -1059222720%
060104 193628 urls\urls.txt:0+94
060104 193629 map -1059281472%
060104 193629 urls\urls.txt:0+94
060104 193630 map -1059340416%
060104 193630 urls\urls.txt:0+94
060104 193631 map -1059400384%
060104 193631 urls\urls.txt:0+94
060104 193632 map -1059459584%
060104 193632 urls\urls.txt:0+94
060104 193633 map -1059518720%
060104 193633 urls\urls.txt:0+94
060104 193634 map -1059577408%
060104 193634 urls\urls.txt:0+94
060104 193635 map -1059636608%
060104 193635 urls\urls.txt:0+94
060104 193636 map -1059695424%
060104 193636 urls\urls.txt:0+94
060104 193637 map -1059755072%
060104 193637 urls\urls.txt:0+94
060104 193638 map -1059814080%
060104 193638 urls\urls.txt:0+94
060104 193639 map -1059870912%
060104 193639 urls\urls.txt:0+94
060104 193640 map -1059927296%
060104 193640 urls\urls.txt:0+94
060104 193641 map -1059984896%
060104 193641 urls\urls.txt:0+94
060104 193642 map -1060041792%
060104 193642 urls\urls.txt:0+94
060104 193643 map -1060097472%
060104 193643 urls\urls.txt:0+94
060104 193644 map -1060151616%
060104 193644 urls\urls.txt:0+94
060104 193645 map -1060208192%
060104 193645 urls\urls.txt:0+94
060104 193646 map -1060262080%
060104 193646 urls\urls.txt:0+94
060104 193647 map -1060319424%
060104 193647 urls\urls.txt:0+94
060104 193648 map -1060375168%
060104 193648 urls\urls.txt:0+94
060104 193649 map -1060429824%
060104 193649 urls\urls.txt:0+94
060104 193650 map -1060485312%
060104 193651 urls\urls.txt:0+94
060104 193651 map -1060542912%
060104 193652 urls\urls.txt:0+94
060104 193652 map -1060596096%
060104 193653 urls\urls.txt:0+94
060104 193653 map -1060653568%
060104 193654 urls\urls.txt:0+94
060104 193654 map -1060710528%
060104 193655 urls\urls.txt:0+94
060104 193655 map -1060761920%
060104 193656 urls\urls.txt:0+94
060104 193656 map -1060817216%
060104 193657 urls\urls.txt:0+94
060104 193657 map -1060872704%
060104 193658 urls\urls.txt:0+94
060104 193658 map -1060930112%
060104 193659 urls\urls.txt:0+94
060104 193659 map -1060989184%
060104 193700 urls\urls.txt:0+94
060104 193700 map -1061046784%
060104 193701 urls\urls.txt:0+94
060104 193701 map -1061105088%
060104 193702 urls\urls.txt:0+94
060104 193702 map -1061162304%
060104 193703 urls\urls.txt:0+94
060104 193703 map -1061218112%
060104 193704 urls\urls.txt:0+94
060104 193704 map -1061272512%
060104 193705 urls\urls.txt:0+94
060104 193705 map -1061322816%
060104 193706 urls\urls.txt:0+94
060104 193706 map -1061357568%
060104 193707 urls\urls.txt:0+94
060104 193707 map -1061399488%
060104 193708 urls\urls.txt:0+94
060104 193708 map -1061437120%
060104 193709 urls\urls.txt:0+94
060104 193709 map -1061479680%
060104 193710 urls\urls.txt:0+94
060104 193710 map -1061515328%
060104 193711 urls\urls.txt:0+94
060104 193711 map -1061560384%
060104 193712 urls\urls.txt:0+94
060104 193712 map -1061619200%
060104 193713 urls\urls.txt:0+94
060104 193713 map -1061668288%
060104 193714 urls\urls.txt:0+94
060104 193714 map -1061727808%
060104 193715 urls\urls.txt:0+94
060104 193715 map -1061785920%
060104 193716 urls\urls.txt:0+94
060104 193717 urls\urls.txt:0+94
060104 193717 map -1061886016%
060104 193718 urls\urls.txt:0+94
060104 193718 map -1061944512%
060104 193719 urls\urls.txt:0+94
060104 193719 map -1061997504%
060104 193720 urls\urls.txt:0+94
060104 193720 map -1062055424%
060104 193721 urls\urls.txt:0+94
060104 193721 map -1062108480%
060104 193722 urls\urls.txt:0+94
060104 193723 map -1062164928%
060104 193723 urls\urls.txt:0+94
060104 193724 map -1062223232%
060104 193724 urls\urls.txt:0+94
060104 193725 map -1062277120%
060104 193725 urls\urls.txt:0+94
060104 193726 map -1062324480%
060104 193726 urls\urls.txt:0+94
060104 193727 map -1062380608%
060104 193727 urls\urls.txt:0+94
060104 193728 map -1062435584%
060104 193728 urls\urls.txt:0+94
060104 193729 map -1062489280%
060104 193729 urls\urls.txt:0+94
060104 193730 map -1062541568%
060104 193730 urls\urls.txt:0+94
060104 193731 map -1062601024%
060104 193731 urls\urls.txt:0+94
060104 193732 map -1062658688%
060104 193732 urls\urls.txt:0+94
060104 193733 map -1062716928%
060104 193733 urls\urls.txt:0+94
060104 193734 map -1062774784%
060104 193734 urls\urls.txt:0+94
060104 193735 map -1062820480%
060104 193735 urls\urls.txt:0+94
060104 193736 map -1062878272%
060104 193736 urls\urls.txt:0+94
060104 193737 map -1062938112%
060104 193737 urls\urls.txt:0+94
060104 193738 map -1062993728%
060104 193738 urls\urls.txt:0+94
060104 193739 map -1063050304%
060104 193739 urls\urls.txt:0+94
060104 193740 map -1063104896%
060104 193740 urls\urls.txt:0+94
060104 193741 map -1063163584%
060104 193741 urls\urls.txt:0+94
060104 193742 map -1063222784%
060104 193742 urls\urls.txt:0+94
060104 193743 map -1063280320%
060104 193743 urls\urls.txt:0+94
060104 193744 map -1063337984%
060104 193744 urls\urls.txt:0+94
060104 193745 map -1063386816%
060104 193745 urls\urls.txt:0+94
060104 193746 map -1063445632%
060104 193746 urls\urls.txt:0+94
060104 193747 map -1063506112%
060104 193747 urls\urls.txt:0+94
060104 193748 map -1063563712%
060104 193748 urls\urls.txt:0+94
060104 193749 map -1063621504%
060104 193749 urls\urls.txt:0+94
060104 193750 map -1063680320%
060104 193750 urls\urls.txt:0+94
060104 193751 map -1063738368%
060104 193751 urls\urls.txt:0+94
060104 193752 map -1063794176%
060104 193752 urls\urls.txt:0+94
060104 193753 map -1063850688%
060104 193753 urls\urls.txt:0+94
060104 193754 map -1063903104%
060104 193754 urls\urls.txt:0+94
060104 193755 map -1063961408%
060104 193755 urls\urls.txt:0+94
060104 193756 map -1064011072%
060104 193756 urls\urls.txt:0+94
060104 193757 map -1064064704%
060104 193757 urls\urls.txt:0+94
060104 193758 map -1064122112%
060104 193758 urls\urls.txt:0+94
060104 193759 map -1064180480%
060104 193759 urls\urls.txt:0+94
060104 193800 map -1064235200%
060104 193800 urls\urls.txt:0+94
060104 193801 map -1064287296%
060104 193801 urls\urls.txt:0+94
060104 193802 map -1064347392%
060104 193802 urls\urls.txt:0+94
060104 193803 map -1064405120%
060104 193804 urls\urls.txt:0+94
060104 193804 map -1064460800%
060104 193805 urls\urls.txt:0+94
060104 193805 map -1064513408%
060104 193806 urls\urls.txt:0+94
060104 193806 map -1064557312%
060104 193807 urls\urls.txt:0+94
060104 193807 map -1064603072%
060104 193808 urls\urls.txt:0+94
060104 193808 map -1064643904%
060104 193809 urls\urls.txt:0+94
060104 193809 map -1064692224%
060104 193810 urls\urls.txt:0+94
060104 193810 map -1064735232%
060104 193811 urls\urls.txt:0+94
060104 193811 map -1064791296%
060104 193812 urls\urls.txt:0+94
060104 193812 map -1064850816%
060104 193813 urls\urls.txt:0+94
060104 193813 map -1064910528%
060104 193814 urls\urls.txt:0+94
060104 193814 map -1064969984%
060104 193815 urls\urls.txt:0+94
060104 193815 map -1065028480%
060104 193816 urls\urls.txt:0+94
060104 193816 map -1065087616%
060104 193817 urls\urls.txt:0+94
060104 193817 map -1065148032%
060104 193818 urls\urls.txt:0+94
060104 193818 map -1065207872%
060104 193819 urls\urls.txt:0+94
Rozina Sorathia,
rozinas@kpitcummins.com