Run the “Selenium Webdriver” in the mapreduce frame of Hadoop, freezed in the map step.

[chenchang <cc...@qq.com>]

Hello,
I'm sorry to disturb you. Few days ago, I
decided to let
selenium webdriver(a third-party package)
run in mapreduce frame of hadoop.
And I met a problem. The map step freeze
in "new FirefoxDriver();". The FirefoxDriver
class was in the third-party jar named
"selenium-server-standalone-2.38.0.jar".
If someone had some experiences or
interest in, I need your help ! ^_^.
Thanks a lot.
Some Details:
1. environment
ubuntu 10.04 32bit;
hadoop-1.2.1;
Mozilla Firefox 17.0.5;
selenium-server-standalone-2.38.0.jar;
Xvfb;
2.  tips
(1)hadoop run in Pesudo-distributed mode
(2)Every thing is ok when I run the code
in Eclipse.(I will show the demo code lastly)
(3) org.openqa.selenium.WebDriverException: Unable to bind to locking port 7054 within 45000 ms
use commad "ps -A | grep firefox"
if some firefox was setup, use"killall firefox"
(4) Let the code run in command line
Maybe you would meet "Error no display specified"
You can install "xvfb"
and setup xvfb using "Xvfb :99 -ac 2>/dev/null &"
append a line "export DISPLAY=:99" to the end of
HADOOP_HOME/conf/hadoop-env.sh
3.  problem details
In order to run the code in command line,
I use "Xvfb" to stop the Firefox graphic
interface. Then the problem I said in the
beginning arise.
I go through the tasktraker's log and find
the code freeze at
"this.driver = new FirefoxDriver(ffprofile);"
Although the code freezed, the firefox
has been setup, I check it using
"ps -A | grep firefox"


4. code demo
public class MapRunnerNewFirefox extends Configured implements Tool,
MapRunnable<LongWritable, Text, Text, Text>{


    public static final Logger LOG = LoggerFactory.getLogger(MapRunnerNewFirefox.class);


    @Override
    public void configure(JobConf conf) {
    }
    @Override
    public void run(RecordReader<LongWritable, Text> recordReader,
        OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
        LongWritable key = new LongWritable(-1);// shouldn't be null ,otherwise the recordReader will report nullpointer err;
        Text val = new Text("begin text");		// same as up line;


	int i = 0;
	reporter.progress();
	while(recordReader.next(key, val)){			
	    if(LOG.isInfoEnabled()){
	        LOG.info("key: "+key.toString()+" val: "+val.toString());
	    }
			
	    String temp = "ao";
	    NewFirefox ff = new NewFirefox("/home/cc/firefox/firefox/firefox");
	    output.collect(new Text("get-"+key.toString()),
					new Text(temp));
	}
    }


	@Override
    public int run(String[] args) throws Exception {
	if(LOG.isInfoEnabled()) {
		LOG.info("set maprunner conf");
	}
	Path urlDir = new Path(args[0]);
	Path resultDir = new Path(args[1] + System.currentTimeMillis());
	JobConf job = new JobConf(getConf());


	job.setNumMapTasks(1);
	job.setJobName("hello maprunners");
		
	job.setInputFormat(TextInputFormat.class);
	FileInputFormat.addInputPath(job, urlDir);
		
	job.setMapRunnerClass(MapRunnerNewFirefox.class);
		
	FileOutputFormat.setOutputPath(job, resultDir);
	job.setOutputFormat(TextOutputFormat.class);
	job.setOutputKeyClass(Text.class);
	job.setOutputValueClass(Text.class);
		
	JobClient.runJob(job);		
	return 0;
    }
    public static void main(String[] args) throws Exception{	
	Configuration conf = new Configuration();		
	int res = ToolRunner.run(conf, new MapRunnerNewFirefox(), args);
	System.exit(res);	
    }
}


public class NewFirefox {
    private WebDriver driver;
    private static final Logger LOG = LoggerFactory.getLogger(NewFirefox.class);
	
    public NewFirefox(String firefoxPath){
	if(LOG.isInfoEnabled()){
		LOG.info("firefox ****0");
	}
	System.setProperty("webdriver.firefox.bin",
			firefoxPath);
	if(LOG.isInfoEnabled()){
		LOG.info("firefox ****1");
	}
	ProfilesIni profile = new ProfilesIni();
	FirefoxProfile ffprofile = profile.getProfile("default");
	if(LOG.isInfoEnabled()){
		LOG.info("firefox ****2");
	}
	this.driver = new FirefoxDriver(ffprofile);
	if(LOG.isInfoEnabled()){
		LOG.info("firefox ****3");
	}
	this.driver.quit();
	if(LOG.isInfoEnabled()){
		LOG.info("firefox quit");
	}
    }
}




Thanks a lot
-chenchang


.