[jira] [Created] (SQOOP-1301) 'run' method in Sqoop.java is not thread-safe

["Ajay Chitre (JIRA)" <ji...@apache.org>]

Ajay Chitre created SQOOP-1301:
----------------------------------

             Summary: 'run' method in Sqoop.java is not thread-safe
                 Key: SQOOP-1301
                 URL: https://issues.apache.org/jira/browse/SQOOP-1301
             Project: Sqoop
          Issue Type: Bug
    Affects Versions: 1.4.4
            Reporter: Ajay Chitre
             Fix For: 1.4.5


It seems the ‘run’ method in Sqoop is not thread-safe.  When multiple Sqoop jobs are triggered exactly at the same time, they end up stepping on each other.  Here’s what seems to be the problem:  The ‘run’ method calls tool.parseArguments which in turn calls ConfigurationHelper.parseGenericOptions which in turn creates a new instance of GenericOptionsParser.  The constructor for GenericOptionsParser is not thread-safe because it ends up calling ‘buildGeneralOptions’ which uses OptionBuilder.withArgName.  This method uses instance variables thereby making it thread unsafe.

The way we’ve got around it is by creating a ‘Lock’ object.  This seems to be working.  If there’s a better way, please let us know.  If not, please consider adding this feature.  We can create a patch if there’s an interest.  Thanks.

Ajay Chitre (achitre@cisco.com)
Virendra Singh (virsingh@cisco.com)


Anyway, here’s what we’ve done:

1)  Created a class that extends Sqoop.java
public class DlSqoop extends Sqoop {

2)  Created a Lock object:

    private static Lock monitor = new ReentrantLock();

3)  Overridden ‘run’ method as follows:

  public int run(String [] args) {
    if (options.getConf() == null) {
      // Configuration wasn't initialized until after the ToolRunner
      // got us to this point. ToolRunner gave Sqoop itself a Conf
      // though.
      options.setConf(getConf());
    }

    try {
      monitor.lock();
      options = tool.parseArguments(args, null, options, false);
      tool.appendArgs(this.childPrgmArgs);
      tool.validateOptions(options);
    } catch (Exception e) {
      // Couldn't parse arguments.
      // Log the stack trace for this exception
      LOG.debug(e.getMessage(), e);
      // Print exception message.
      System.err.println(e.getMessage());
      // Print the tool usage message and exit.
      ToolOptions toolOpts = new ToolOptions();
      tool.configureOptions(toolOpts);
      tool.printHelp(toolOpts);
      return 1; // Exit on exception here.
       } finally {
            monitor.unlock();
        }
    return tool.run(options);
  }




--
This message was sent by Atlassian JIRA
(v6.2#6252)