[jira] [Commented] (SQOOP-453) Oozie does not work with Sqoop metastore

["Cheolsoo Park (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260113#comment-13260113 ] 

Cheolsoo Park commented on SQOOP-453:
-------------------------------------

I did some digging on this, and I was able to reproduce the error as follows:

1) Disable sqoop.metastore.client.record.password in sqoop-site.xml so that password is not saved in the metastore.

{code}
  <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>false</value>
    <description>If true, allow saved passwords in the metastore.
    </description>
  </property>
{code}

2) Save some job in the metastore. For example,

{code}
sqoop job --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop --create mysqlimport -- import --connect jdbc:mysql://localhost/sqooptestdb --username sqooptest --password ***** --table foo -m 1
{code}

3) Run the saved job via Oozie, where my workflow.xml is something like this:

{code}
<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">
    ...
    <action name="sqoop-node">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            ...
            <command>job --meta-connect jdbc:hsqldb:hsql://localhost:16000/sqoop --exec mysqlimport</command>
        </sqoop>
        ...
    </action>
    ...
</workflow-app>
{code}

4) This job crashes with a NPE as shown in the description.

The problem is, with sqoop.metastore.client.record.password = false, Sqoop prompts the user to enter the password when executing a saved job. However, it seems that Oozie task processes are not given stdin, resulting in a NPE at the following line in SqoopOptions.java:

{code}
return new String(System.console().readPassword("Enter password: ")); // System.console() is null.
{code}

A workaround for this problem would be setting sqoop.metastore.client.record.password to true. But apparently, this will make the metastore insecure.

Does anyone have a better suggestion?
                
> Oozie does not work with Sqoop metastore
> ----------------------------------------
>
>                 Key: SQOOP-453
>                 URL: https://issues.apache.org/jira/browse/SQOOP-453
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.3.0
>            Reporter: Patrick Angeles
>
> The sqoop metastore is handy for retaining lastvalue during incremental imports.
> Unfortunately, it does not work with Oozie. Here's the log from the launcher:
> java.lang.NullPointerException
> 	at com.cloudera.sqoop.SqoopOptions.securePasswordEntry(SqoopOptions.java:1069)
> 	at com.cloudera.sqoop.SqoopOptions.setPasswordFromConsole(SqoopOptions.java:1077)
> 	at com.cloudera.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:562)
> 	at com.cloudera.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:303)
> 	at com.cloudera.sqoop.tool.JobTool.execJob(JobTool.java:203)
> 	at com.cloudera.sqoop.tool.JobTool.run(JobTool.java:288)
> 	at com.cloudera.sqoop.Sqoop.run(Sqoop.java:146)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> 	at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:182)
> 	at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:221)
> 	at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:230)
> 	at com.cloudera.sqoop.Sqoop.main(Sqoop.java:239)
> 	at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:198)
> 	at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:172)
> 	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:26)
> 	at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:43)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:391)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:264)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira