Custom Options for User defined plugins for sqoop

[Rakesh Sharma <ra...@expedia.com>]

Hi,

Sqoop is based on Hadoop Tools infrastructure, which allows users to develop their own plugins for sqoop and extend sqoop as per their requirements without changing the base sqoop. Users can develop their plugins as separate jars, deploy them in $SQOOP_LIB and register with sqoop as :

<property>
    <name>sqoop.tool.plugins</name>
    <value>com.expedia.sqoop.tool.HiveMergePlugin</value>
    <description>A comma-delimited list of ToolPlugin implementations
      which are consulted, in order, to register SqoopTool instances which
      allow third-party tools to be used.
    </description>
</property>

This functionality is already there and we are able to use this develop our own plugins. Infact there are a couple of teams in Expedia which are using their own plugins with sqoop. I am sure there might be others who will be using this or willing to use this functionality.. Our Team have also a tool called hive-merge for sqoop which merges two hive tables,  but this tool needs to use some option names which are not available in SqoopOptions. The limitation with user defined plugin is that it doesn't support custom defined options as all the sqoop options for sqoop are tied to names and types in SqoopOptions. We have made a small enhancement which adds a map member to SqoopOptions. This is a very small change but it opens up Sqoop for user defined tools with custom options. It will open up sqoop for user defined plugins. We have also added support for its persistence into metastore. The patch is already available for the issue :
https://issues.apache.org/jira/browse/SQOOP-2333

The review request for the fix is available at :
https://reviews.apache.org/r/36042/

Although the change is small, it will be a great addition to Sqoop and will enable developers to develop their own plugins with user defined options and contribute back to the community without knowing much details of sqoop. We have also added documentation about how to develop user defined plugins and custom options for user defined plugins.

Please review and suggest.

Thanks and Regards,
Rakesh.