You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Nagarjun Pola <np...@usc.edu> on 2015/02/20 06:44:43 UTC
Problem Fetching with Selenium Installed
I get the following error when tried with selenium. Firefox pops up couple
of times but fetches nothing.
Can anyone help me on this issue?
*-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1*
** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
* maxThreads = 1*
* inProgress = 1*
* crawlDelay = 5000*
* minCrawlDelay = 0*
* nextFetchTime = 1424410799976*
* now = 1424410803146*
* 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
*fetch of
http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
<http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
failed with: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
*-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
fetchQueues.getQueueCount=1*
** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
* maxThreads = 1*
* inProgress = 0*
* crawlDelay = 5000*
* minCrawlDelay = 0*
* nextFetchTime = 1424410808305*
* now = 1424410804147*
Re: Problem Fetching with Selenium Installed
Posted by Qing Liu <li...@usc.edu>.
Hi Nagarijun, I'm really confused that since we are using virtual
framebuffer, why would Firefox pop out? I started Xvfb, then opened firefox
in terminal and nothing popped out, but firefox is running.
On Thu, Feb 19, 2015 at 10:58 PM, Nagarjun Pola <np...@usc.edu> wrote:
> Thank You Mohammed.
>
> I just got a fresh copy of Nutch and buit everything again from scratch
> and it seems to be fetching lot of data with Firefox popping every now and
> then.
>
> Best,
> Nagarjun Pola
>
>
> On Thu, Feb 19, 2015 at 10:51 PM, Mohammad Al-Mohsin <me...@mem9.net> wrote:
>
>> Hi Nagarjun,
>>
>> I faced the same issue and got it resolved by deleting 'runtime'
>> directory and then recompiling Nutch (along with Selenium plugin).
>>
>> So cd into nutch trunk or branch and then execute:
>>
>> rm -r runtime
>> ant runtime
>>
>> Make sure you take a backup of your Nutch configurations before deleting
>> runtime directory.
>>
>> Best regards,
>> Mohammad Al-Mohsin
>>
>> On Thu, Feb 19, 2015 at 10:16 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>
>>> Yes. I should do that.
>>>
>>> Thank You Jiaxin.
>>>
>>> Best,
>>> Nagarjun Pola
>>>
>>>
>>> On Thu, Feb 19, 2015 at 10:15 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>>>
>>>> Hmm...Why dont you try to git clone a new nutch and then use the nutch
>>>> only to see if you can crawl or not?
>>>>
>>>> On Thu, Feb 19, 2015 at 10:09 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>>>
>>>>> Hmm I don't think the crawler is being blocked of politeness because I
>>>>> am using the default Nutch configuration which is 1 request per second.
>>>>>
>>>>> And when I try to crawl with the sample URL by disabling Nutch plugin
>>>>> in the Nutch-site.xml I can retrieve some links.
>>>>>
>>>>> The problem seems to be in the selenium plugin. Though Firefox pops
>>>>> nothing is fetched.
>>>>>
>>>>> Best,
>>>>> Nagarjun Pola
>>>>>
>>>>>
>>>>> On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>>>>>
>>>>>> Hi, my teammate is also suffering from this situation now and I
>>>>>> encountered this situation last night. But I am able to crawl now almost
>>>>>> without doing anything. The reason I may guess is that your crawler is
>>>>>> blocked by the website because not being polite. At least I believe that's
>>>>>> the reason why I got the same *Could not initialize class
>>>>>> org.apache.http.impl.conn.* last night. I don't how to solve it,
>>>>>> though..... Fortunate enough I think I am unbanned now, I guess? Hope it
>>>>>> helps......
>>>>>>
>>>>>> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>>>>>
>>>>>>> I get the following error when tried with selenium. Firefox pops
>>>>>>> up couple of times but fetches nothing.
>>>>>>>
>>>>>>> Can anyone help me on this issue?
>>>>>>>
>>>>>>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
>>>>>>> fetchQueues.getQueueCount=1*
>>>>>>>
>>>>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>>>>
>>>>>>> * maxThreads = 1*
>>>>>>>
>>>>>>> * inProgress = 1*
>>>>>>>
>>>>>>> * crawlDelay = 5000*
>>>>>>>
>>>>>>> * minCrawlDelay = 0*
>>>>>>>
>>>>>>> * nextFetchTime = 1424410799976*
>>>>>>>
>>>>>>> * now = 1424410803146*
>>>>>>>
>>>>>>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>>>>>>>
>>>>>>> *fetch of
>>>>>>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
>>>>>>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
>>>>>>> failed with: java.lang.NoClassDefFoundError: Could not initialize class
>>>>>>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>>>>>>>
>>>>>>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
>>>>>>> fetchQueues.getQueueCount=1*
>>>>>>>
>>>>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>>>>
>>>>>>> * maxThreads = 1*
>>>>>>>
>>>>>>> * inProgress = 0*
>>>>>>>
>>>>>>> * crawlDelay = 5000*
>>>>>>>
>>>>>>> * minCrawlDelay = 0*
>>>>>>>
>>>>>>> * nextFetchTime = 1424410808305*
>>>>>>>
>>>>>>> * now = 1424410804147*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problem Fetching with Selenium Installed
Posted by Nagarjun Pola <np...@usc.edu>.
Thank You Mohammed.
I just got a fresh copy of Nutch and buit everything again from scratch and it seems to be fetching lot of data with Firefox popping every now and then.
Best,
Nagarjun Pola
On Thu, Feb 19, 2015 at 10:51 PM, Mohammad Al-Mohsin <me...@mem9.net> wrote:
> Hi Nagarjun,
> I faced the same issue and got it resolved by deleting 'runtime' directory
> and then recompiling Nutch (along with Selenium plugin).
> So cd into nutch trunk or branch and then execute:
> rm -r runtime
> ant runtime
> Make sure you take a backup of your Nutch configurations before deleting
> runtime directory.
> Best regards,
> Mohammad Al-Mohsin
> On Thu, Feb 19, 2015 at 10:16 PM, Nagarjun Pola <np...@usc.edu> wrote:
>> Yes. I should do that.
>>
>> Thank You Jiaxin.
>>
>> Best,
>> Nagarjun Pola
>>
>>
>> On Thu, Feb 19, 2015 at 10:15 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>>
>>> Hmm...Why dont you try to git clone a new nutch and then use the nutch
>>> only to see if you can crawl or not?
>>>
>>> On Thu, Feb 19, 2015 at 10:09 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>>
>>>> Hmm I don't think the crawler is being blocked of politeness because I
>>>> am using the default Nutch configuration which is 1 request per second.
>>>>
>>>> And when I try to crawl with the sample URL by disabling Nutch plugin in
>>>> the Nutch-site.xml I can retrieve some links.
>>>>
>>>> The problem seems to be in the selenium plugin. Though Firefox pops
>>>> nothing is fetched.
>>>>
>>>> Best,
>>>> Nagarjun Pola
>>>>
>>>>
>>>> On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>>>>
>>>>> Hi, my teammate is also suffering from this situation now and I
>>>>> encountered this situation last night. But I am able to crawl now almost
>>>>> without doing anything. The reason I may guess is that your crawler is
>>>>> blocked by the website because not being polite. At least I believe that's
>>>>> the reason why I got the same *Could not initialize class
>>>>> org.apache.http.impl.conn.* last night. I don't how to solve it,
>>>>> though..... Fortunate enough I think I am unbanned now, I guess? Hope it
>>>>> helps......
>>>>>
>>>>> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>>>>
>>>>>> I get the following error when tried with selenium. Firefox pops up
>>>>>> couple of times but fetches nothing.
>>>>>>
>>>>>> Can anyone help me on this issue?
>>>>>>
>>>>>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
>>>>>> fetchQueues.getQueueCount=1*
>>>>>>
>>>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>>>
>>>>>> * maxThreads = 1*
>>>>>>
>>>>>> * inProgress = 1*
>>>>>>
>>>>>> * crawlDelay = 5000*
>>>>>>
>>>>>> * minCrawlDelay = 0*
>>>>>>
>>>>>> * nextFetchTime = 1424410799976*
>>>>>>
>>>>>> * now = 1424410803146*
>>>>>>
>>>>>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>>>>>>
>>>>>> *fetch of
>>>>>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
>>>>>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
>>>>>> failed with: java.lang.NoClassDefFoundError: Could not initialize class
>>>>>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>>>>>>
>>>>>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
>>>>>> fetchQueues.getQueueCount=1*
>>>>>>
>>>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>>>
>>>>>> * maxThreads = 1*
>>>>>>
>>>>>> * inProgress = 0*
>>>>>>
>>>>>> * crawlDelay = 5000*
>>>>>>
>>>>>> * minCrawlDelay = 0*
>>>>>>
>>>>>> * nextFetchTime = 1424410808305*
>>>>>>
>>>>>> * now = 1424410804147*
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Problem Fetching with Selenium Installed
Posted by Mohammad Al-Mohsin <me...@mem9.net>.
Hi Nagarjun,
I faced the same issue and got it resolved by deleting 'runtime' directory
and then recompiling Nutch (along with Selenium plugin).
So cd into nutch trunk or branch and then execute:
rm -r runtime
ant runtime
Make sure you take a backup of your Nutch configurations before deleting
runtime directory.
Best regards,
Mohammad Al-Mohsin
On Thu, Feb 19, 2015 at 10:16 PM, Nagarjun Pola <np...@usc.edu> wrote:
> Yes. I should do that.
>
> Thank You Jiaxin.
>
> Best,
> Nagarjun Pola
>
>
> On Thu, Feb 19, 2015 at 10:15 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>
>> Hmm...Why dont you try to git clone a new nutch and then use the nutch
>> only to see if you can crawl or not?
>>
>> On Thu, Feb 19, 2015 at 10:09 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>
>>> Hmm I don't think the crawler is being blocked of politeness because I
>>> am using the default Nutch configuration which is 1 request per second.
>>>
>>> And when I try to crawl with the sample URL by disabling Nutch plugin in
>>> the Nutch-site.xml I can retrieve some links.
>>>
>>> The problem seems to be in the selenium plugin. Though Firefox pops
>>> nothing is fetched.
>>>
>>> Best,
>>> Nagarjun Pola
>>>
>>>
>>> On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>>>
>>>> Hi, my teammate is also suffering from this situation now and I
>>>> encountered this situation last night. But I am able to crawl now almost
>>>> without doing anything. The reason I may guess is that your crawler is
>>>> blocked by the website because not being polite. At least I believe that's
>>>> the reason why I got the same *Could not initialize class
>>>> org.apache.http.impl.conn.* last night. I don't how to solve it,
>>>> though..... Fortunate enough I think I am unbanned now, I guess? Hope it
>>>> helps......
>>>>
>>>> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>>>
>>>>> I get the following error when tried with selenium. Firefox pops up
>>>>> couple of times but fetches nothing.
>>>>>
>>>>> Can anyone help me on this issue?
>>>>>
>>>>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
>>>>> fetchQueues.getQueueCount=1*
>>>>>
>>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>>
>>>>> * maxThreads = 1*
>>>>>
>>>>> * inProgress = 1*
>>>>>
>>>>> * crawlDelay = 5000*
>>>>>
>>>>> * minCrawlDelay = 0*
>>>>>
>>>>> * nextFetchTime = 1424410799976*
>>>>>
>>>>> * now = 1424410803146*
>>>>>
>>>>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>>>>>
>>>>> *fetch of
>>>>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
>>>>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
>>>>> failed with: java.lang.NoClassDefFoundError: Could not initialize class
>>>>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>>>>>
>>>>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
>>>>> fetchQueues.getQueueCount=1*
>>>>>
>>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>>
>>>>> * maxThreads = 1*
>>>>>
>>>>> * inProgress = 0*
>>>>>
>>>>> * crawlDelay = 5000*
>>>>>
>>>>> * minCrawlDelay = 0*
>>>>>
>>>>> * nextFetchTime = 1424410808305*
>>>>>
>>>>> * now = 1424410804147*
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Problem Fetching with Selenium Installed
Posted by Nagarjun Pola <np...@usc.edu>.
Yes. I should do that.
Thank You Jiaxin.
Best,
Nagarjun Pola
On Thu, Feb 19, 2015 at 10:15 PM, Jiaxin Ye <ji...@usc.edu> wrote:
> Hmm...Why dont you try to git clone a new nutch and then use the nutch only
> to see if you can crawl or not?
> On Thu, Feb 19, 2015 at 10:09 PM, Nagarjun Pola <np...@usc.edu> wrote:
>> Hmm I don't think the crawler is being blocked of politeness because I am
>> using the default Nutch configuration which is 1 request per second.
>>
>> And when I try to crawl with the sample URL by disabling Nutch plugin in
>> the Nutch-site.xml I can retrieve some links.
>>
>> The problem seems to be in the selenium plugin. Though Firefox pops
>> nothing is fetched.
>>
>> Best,
>> Nagarjun Pola
>>
>>
>> On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>>
>>> Hi, my teammate is also suffering from this situation now and I
>>> encountered this situation last night. But I am able to crawl now almost
>>> without doing anything. The reason I may guess is that your crawler is
>>> blocked by the website because not being polite. At least I believe that's
>>> the reason why I got the same *Could not initialize class
>>> org.apache.http.impl.conn.* last night. I don't how to solve it,
>>> though..... Fortunate enough I think I am unbanned now, I guess? Hope it
>>> helps......
>>>
>>> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>>
>>>> I get the following error when tried with selenium. Firefox pops up
>>>> couple of times but fetches nothing.
>>>>
>>>> Can anyone help me on this issue?
>>>>
>>>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
>>>> fetchQueues.getQueueCount=1*
>>>>
>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>
>>>> * maxThreads = 1*
>>>>
>>>> * inProgress = 1*
>>>>
>>>> * crawlDelay = 5000*
>>>>
>>>> * minCrawlDelay = 0*
>>>>
>>>> * nextFetchTime = 1424410799976*
>>>>
>>>> * now = 1424410803146*
>>>>
>>>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>>>>
>>>> *fetch of
>>>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
>>>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
>>>> failed with: java.lang.NoClassDefFoundError: Could not initialize class
>>>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>>>>
>>>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
>>>> fetchQueues.getQueueCount=1*
>>>>
>>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>>
>>>> * maxThreads = 1*
>>>>
>>>> * inProgress = 0*
>>>>
>>>> * crawlDelay = 5000*
>>>>
>>>> * minCrawlDelay = 0*
>>>>
>>>> * nextFetchTime = 1424410808305*
>>>>
>>>> * now = 1424410804147*
>>>>
>>>
>>>
>>
Re: Problem Fetching with Selenium Installed
Posted by Jiaxin Ye <ji...@usc.edu>.
Hmm...Why dont you try to git clone a new nutch and then use the nutch only
to see if you can crawl or not?
On Thu, Feb 19, 2015 at 10:09 PM, Nagarjun Pola <np...@usc.edu> wrote:
> Hmm I don't think the crawler is being blocked of politeness because I am
> using the default Nutch configuration which is 1 request per second.
>
> And when I try to crawl with the sample URL by disabling Nutch plugin in
> the Nutch-site.xml I can retrieve some links.
>
> The problem seems to be in the selenium plugin. Though Firefox pops
> nothing is fetched.
>
> Best,
> Nagarjun Pola
>
>
> On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <ji...@usc.edu> wrote:
>
>> Hi, my teammate is also suffering from this situation now and I
>> encountered this situation last night. But I am able to crawl now almost
>> without doing anything. The reason I may guess is that your crawler is
>> blocked by the website because not being polite. At least I believe that's
>> the reason why I got the same *Could not initialize class
>> org.apache.http.impl.conn.* last night. I don't how to solve it,
>> though..... Fortunate enough I think I am unbanned now, I guess? Hope it
>> helps......
>>
>> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
>>
>>> I get the following error when tried with selenium. Firefox pops up
>>> couple of times but fetches nothing.
>>>
>>> Can anyone help me on this issue?
>>>
>>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
>>> fetchQueues.getQueueCount=1*
>>>
>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>
>>> * maxThreads = 1*
>>>
>>> * inProgress = 1*
>>>
>>> * crawlDelay = 5000*
>>>
>>> * minCrawlDelay = 0*
>>>
>>> * nextFetchTime = 1424410799976*
>>>
>>> * now = 1424410803146*
>>>
>>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>>>
>>> *fetch of
>>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
>>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
>>> failed with: java.lang.NoClassDefFoundError: Could not initialize class
>>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>>>
>>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
>>> fetchQueues.getQueueCount=1*
>>>
>>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>>
>>> * maxThreads = 1*
>>>
>>> * inProgress = 0*
>>>
>>> * crawlDelay = 5000*
>>>
>>> * minCrawlDelay = 0*
>>>
>>> * nextFetchTime = 1424410808305*
>>>
>>> * now = 1424410804147*
>>>
>>
>>
>
Re: Problem Fetching with Selenium Installed
Posted by Nagarjun Pola <np...@usc.edu>.
Hmm I don't think the crawler is being blocked of politeness because I am using the default Nutch configuration which is 1 request per second.
And when I try to crawl with the sample URL by disabling Nutch plugin in the Nutch-site.xml I can retrieve some links.
The problem seems to be in the selenium plugin. Though Firefox pops nothing is fetched.
Best,
Nagarjun Pola
On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <ji...@usc.edu> wrote:
> Hi, my teammate is also suffering from this situation now and I encountered
> this situation last night. But I am able to crawl now almost without doing
> anything. The reason I may guess is that your crawler is blocked by the
> website because not being polite. At least I believe that's the reason why
> I got the same *Could not initialize class org.apache.http.impl.conn.* last
> night. I don't how to solve it, though..... Fortunate enough I think I am
> unbanned now, I guess? Hope it helps......
> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
>> I get the following error when tried with selenium. Firefox pops up couple
>> of times but fetches nothing.
>>
>> Can anyone help me on this issue?
>>
>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
>> fetchQueues.getQueueCount=1*
>>
>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>
>> * maxThreads = 1*
>>
>> * inProgress = 1*
>>
>> * crawlDelay = 5000*
>>
>> * minCrawlDelay = 0*
>>
>> * nextFetchTime = 1424410799976*
>>
>> * now = 1424410803146*
>>
>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>>
>> *fetch of
>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
>> failed with: java.lang.NoClassDefFoundError: Could not initialize class
>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>>
>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
>> fetchQueues.getQueueCount=1*
>>
>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>>
>> * maxThreads = 1*
>>
>> * inProgress = 0*
>>
>> * crawlDelay = 5000*
>>
>> * minCrawlDelay = 0*
>>
>> * nextFetchTime = 1424410808305*
>>
>> * now = 1424410804147*
>>
Re: Problem Fetching with Selenium Installed
Posted by Jiaxin Ye <ji...@usc.edu>.
Hi, my teammate is also suffering from this situation now and I encountered
this situation last night. But I am able to crawl now almost without doing
anything. The reason I may guess is that your crawler is blocked by the
website because not being polite. At least I believe that's the reason why
I got the same *Could not initialize class org.apache.http.impl.conn.* last
night. I don't how to solve it, though..... Fortunate enough I think I am
unbanned now, I guess? Hope it helps......
On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <np...@usc.edu> wrote:
> I get the following error when tried with selenium. Firefox pops up couple
> of times but fetches nothing.
>
> Can anyone help me on this issue?
>
> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1,
> fetchQueues.getQueueCount=1*
>
> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>
> * maxThreads = 1*
>
> * inProgress = 1*
>
> * crawlDelay = 5000*
>
> * minCrawlDelay = 0*
>
> * nextFetchTime = 1424410799976*
>
> * now = 1424410803146*
>
> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>*
>
> *fetch of
> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0
> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0>
> failed with: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory*
>
> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1,
> fetchQueues.getQueueCount=1*
>
> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>*
>
> * maxThreads = 1*
>
> * inProgress = 0*
>
> * crawlDelay = 5000*
>
> * minCrawlDelay = 0*
>
> * nextFetchTime = 1424410808305*
>
> * now = 1424410804147*
>