[jira] [Updated] (SQOOP-2496) Sqoop2: Provide a way to inject external connectors

["Abraham Fine (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SQOOP-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abraham Fine updated SQOOP-2496:
--------------------------------
    Fix Version/s:     (was: 1.99.7)
                   1.99.8

> Sqoop2: Provide a way to inject external connectors
> ---------------------------------------------------
>
>                 Key: SQOOP-2496
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2496
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.99.6
>            Reporter: Jarek Jarcec Cecho
>             Fix For: 1.99.8
>
>         Attachments: MapreduceSubmissionEngine.java
>
>
> On internal hackathon we we're hacking Sqoop 2 connector with [~singhashish] and we went through few troubles that we should address.
> We have a a [configuration property|https://github.com/apache/sqoop/blob/sqoop2/dist/src/main/server/conf/sqoop.properties#L173] for extra directory from which we will load jar files. We were able to use this configuration property to load our hacked connector to Server, but we were not able to get it working through job submission. Here is the exception that we hit:
> {code} 
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: jar:file://var/lib/sqoop/connectors/Connector-1.0-SNAPSHOT.jar!
>         at org.apache.hadoop.fs.Path.initialize(Path.java:206)
>         at org.apache.hadoop.fs.Path.<init>(Path.java:172)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:215)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
>         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
>         at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
>         at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
>         at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
>         at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
>         at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
>         at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
>         at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> {code}
> To put into nutshell, [ClassUtils.jarForClass|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/utils/ClassUtils.java#L136] returns for external connectors path starting with prefix {{jar:file}} and suffix {{!}} that breaks mapreduce code. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)