svn commit: r1703944 - in /nutch/trunk: CHANGES.txt src/plugin/protocol-selenium/README.md

[ma...@apache.org]

Author: mattmann
Date: Sat Sep 19 05:27:02 2015
New Revision: 1703944

URL: http://svn.apache.org/viewvc?rev=1703944&view=rev
Log:
Fix for NUTCH-2104: Add documentation to the protocol-selenium plugin Readme file re: selenium grid implementation contributed by Kim Whitehall <ki...@jpl.nasa.gov> this closes #60.

Modified:
    nutch/trunk/CHANGES.txt
    nutch/trunk/src/plugin/protocol-selenium/README.md

Modified: nutch/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/nutch/trunk/CHANGES.txt?rev=1703944&r1=1703943&r2=1703944&view=diff
==============================================================================
--- nutch/trunk/CHANGES.txt (original)
+++ nutch/trunk/CHANGES.txt Sat Sep 19 05:27:02 2015
@@ -2,6 +2,9 @@ Nutch Change Log
    
 Nutch Current Development 1.11-SNAPSHOT
 
+* NUTCH-2104 Add documentation to the protocol-selenium plugin Readme file 
+  re: selenium grid implementation (Kim Whitehall via mattmann)
+
 * NUTCH-2099 Refactoring the REST endpoints for integration with 
   webui (Sujen Shah via mattmann)
 

Modified: nutch/trunk/src/plugin/protocol-selenium/README.md
URL: http://svn.apache.org/viewvc/nutch/trunk/src/plugin/protocol-selenium/README.md?rev=1703944&r1=1703943&r2=1703944&view=diff
==============================================================================
--- nutch/trunk/src/plugin/protocol-selenium/README.md (original)
+++ nutch/trunk/src/plugin/protocol-selenium/README.md Sat Sep 19 05:27:02 2015
@@ -14,7 +14,9 @@ There are essentially two ways in which
 
 # Installation
 
-## Part 1: Setting up Selenium
+## Part 1: 
+
+### A) Setting up Selenium (local mode)
 
  * Ensure that you have Firefox installed. More info about the package @ [launchpad](https://launchpad.net/ubuntu/trusty/+source/firefox)
 ```
@@ -31,9 +33,38 @@ sudo apt-get install xorg synaptic xvfb
 sudo /usr/bin/Xvfb :11 -screen 0 1024x768x24 &
 sudo export DISPLAY=:11
 ```
+### B) Setting up a Selenium Grid 
+
+Using the Selenium Grid will allow you to parallelize the job by facilitating access of several instances of browsers whether on one machine or on several machines. Note that grid facilitates heterogeneity with regards to browser types used. However, these steps have been tested using a homogenous firefox Selenium Grid. 
+
+ * Download the [Selenium Standalone Server](http://www.seleniumhq.org/download/) and follow the installation instructions.
+ 
+ * Some important configurations to note while setting up the selenium-hub and the selenium-nodes are:
+    * For the hub: 
+      - maxSession (how many browser sessions to allow on the grid at a time)
+      - browserTimeout (how long to wait before timing out a browser session. This is dependent on the interactivity to be completed on the page)
+      
+    * For the nodes:
+      - browserName=<browser>, maxInstances (the max number of instances of the same version browser to allow per a system)
+      - browserName=<browser>, maxSession (the max number of sessions of any type of browser/version to allow per a system)
+      
+  * Go headless with your selenium Grid installation. There are different ways to this. See [this resource](http://elementalselenium.com/tips/38-headless) for further details. 
+ 
+  * For Nutch efficiency, and optimization of the grid, consider editing the following configs in **nutch-site.xml**
+    - fetcher.threads.per.queue (change value to the value of the maxSession config on the hub)
+    - fetcher.threads.fetch (change value to the value of the maxSession config on the hub)
+    - fetcher.server.delay (As multiple threads may be accessing a single server at a time, consider changing this value to 4-5 seconds for politeness)
+    - fetcher.server.min.delay (As multiple threads may be accessing a single server at a time, consider changing this values to 4-5 seconds for politeness)
+    - Ensure all configs for the hub mentioned in Part 2 are appropriately set. 
+
+  * To activate the full selenium grid, edit **$NUTCH_HOME/runtime/local/bin/crawl** script:
+    - numThreads = maxSession on nodes * num of nodes
+
+
 ## Part 2: Installing plugin for Nutch (where NUTCH_HOME is the root of your nutch install)
 
  * Ensure that the plugin will be used as the protocol parser in your config
+
 ```
 <!-- NUTCH_HOME/conf/nutch-site.xml -->
 
@@ -53,7 +84,7 @@ sudo export DISPLAY=:11
   </property>
 ```
 
-Then ensure that you have the correct configuration set within the following configuration options
+* Then ensure that you have the correct configuration set within the following configuration options
 
 ```
 <!-- protocol-selenium plugin properties -->
@@ -119,6 +150,22 @@ Then ensure that you have the correct co
   <description>Selenium Hub Location connection protocol</description>
 </property>
 
+<property>
+  <name>selenium.grid.driver</name>
+  <value>firefox</value>
+  <description>A String value representing the flavour of Selenium 
+    WebDriver() used on the selenium grid. Currently the following options
+    exist - 'firefox' </description>
+</property>
+
+<property>
+  <name>selenium.grid.binary</name>
+  <value></value>
+  <description>A String value representing the path to the browser binary 
+    location for each node
+ </description>
+</property>
+
 <!-- lib-selenium configuration -->
 <property>
   <name>libselenium.page.load.delay</name>
@@ -140,4 +187,9 @@ ant runtime
 
  * Start your web crawl (Ensure that you followed the above steps and have started your xvfb display as shown above)
 
+## Part 3: Common Pitfalls
 
+* Be sure your browser version and selenium version are compatible 
+* Be sure to start the Xvfb window then start selenium
+* Disconnecting and reconnect nodes after a hub config change has proven useful in our tests. 
+* Be sure that each browser session deallocates its webdriver resource independently of any other tests running on other broswers (check out driver.quit() and driver.close()).