You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Dian Fu <di...@gmail.com> on 2015/10/23 11:01:42 UTC
Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs
-----
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java 285e893
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 522ed08
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java f49fca3
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java fd2d1b4
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 7c1ec18
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java b7aa8c6
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 0302150
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > My apologies for jumping so late on this review. Few high level notes:
> >
> > 1) It seems that the patch no longer applies cleanly, so you will have to rebase it.
> >
> > 2) I know that we've changed the design quite a bit - would you mind updating the JIRA and the review board summary with what exactly we're trying to accomplish here? I don't think that the title "Sqoop2: Allow connectors to express jar dependencies" is still relevant. It's more about wrap connector's dependencies inside the connector's jar, right?
> >
> > 3) This is rather huge change - I'm wondering if there are independent pieces (like changes to ConnectorClassLoader) that can be submitted via independent JIRA? It will make the subsequent reviews easier.
> >
> > And few more tactical:
Hi Jarcec,
Thanks a lot for the review and suggestions.
1) Have updated the title and description of this JIRA. The new description should can explain what we're trying to do here.
2) Have moved changes related to ConnectorClassLoader to SQOOP-2635.
- Dian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review108543
-----------------------------------------------------------
On Dec. 10, 2015, 9:45 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2015, 9:45 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
> 1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
> 2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
> 3. Load connector class with the connector's own ConnectorClassLoader.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/main/java/org/apache/sqoop/utils/ConnectorClassLoader.java 4c42a78
> common/src/main/resources/org.apache.sqoop.connector-classloader.properties e2936a9
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> pom.xml 91721ce
> server/pom.xml 370a6a2
> shell/src/test/java/org/apache/sqoop/shell/TestCloneCommand.java 7a139ba
> shell/src/test/java/org/apache/sqoop/shell/TestCreateCommand.java 0316a60
> shell/src/test/java/org/apache/sqoop/shell/TestUpdateCommand.java d8f061a
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java, lines 71-81
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131934#file1131934line71>
> >
> > Aren't those dependecies on inside the connector's jar as well?
kitesdk will be inside the connector's jar. For other dependencies such as hadoop, hive, thrift, datanucleus, etc, they will be provided by the hadoop cluster and hive cluster environment separately.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java, lines 74-76
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131940#file1131940line74>
> >
> > Can we abstract this "magic" to standalone method and provide proper test coverage?
> >
> > It seems fragile, so having good tests will hopefully help ensuring that we won't break this in the future.
Good point. Updated.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java, line 55
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131941#file1131941line55>
> >
> > Wondering why do we need to keep the classloader dependencies around? Can't we just store the classloder instance inside the ConnectorHandler or perhaps even easier call connectorInstance.getClass().getClassLoader().
Good point. Updated the patch to use connectorInstance.getClass().getClassLoader() to get the classloader.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > core/src/main/java/org/apache/sqoop/driver/JobManager.java, lines 496-500
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131942#file1131942line496>
> >
> > I'm wondering if this is needed? Based on preliminary research any "new" call inside the initializer() method should use the Connector classes classloader.
For "new" call, that's true that it will use the connector classes classloader. But for "ServiceLoader" api, it will load service classes with the current thread's context class loader by default[1].
[1] https://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html#load(java.lang.Class)
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java, line 86
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131943#file1131943line86>
> >
> > We're not really storing the classpath here, just the jar location. Also I'm wondering if this is really desirable to store since the path is absolute on given machine, right?
1. That's true that it's actually the jar location. Good point, have updated the configuration "JOB_CONNECTOR_FROM_CLASSPATH" to "JOB_CONNECTOR_FROM_LOCATION", also updated "JOB_CONNECTOR_TO_CLASSPATH".
2. When we submit MR job, the connector jar will not be put in the property "tmpjars" of configuration, but in the property "tmpfiles" of configuration as we don't want the connector jar in the classpath of Map/Reduce Job. We want the connector class loaded by the connector's own classloader. So we need a configuration to tell us the connector jar location.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java, lines 62-79
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131946#file1131946line62>
> >
> > I think that easier and more nicer way how to do this is to load the SqoopConnector class with given class loader. Everything else we should "get" for free.
As I said in the above comments, for "ServiceLoader" api, it will load the service classes with the current thread's context class loader by default[1]. So we'd better set the context classloader. Actually we did encountered such cases in kite-sdk[2].
[1] https://docs.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html#load(java.lang.Class)
[2] https://github.com/kite-sdk/kite/blob/master/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/Registration.java
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > server/pom.xml, lines 83-88
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131953#file1131953line83>
> >
> > I'm wondering why do we need the exclusions? They shouldn't be propagated because we marked them as "provided" anyway right?
We want these connectors present but the dependencies of these connectors not present as they have already being packaged in the connector jar. "provided" will make the connector jar also not present, which is not expected.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > common/src/main/resources/org.apache.sqoop.connector-classloader.properties, line 54
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131927#file1131927line54>
> >
> > Wondering what is the "-" at the begging of the line mean?
File "org.apache.sqoop.connector-classloader.properties" defines which classes are consided system classes and should be loaded by the system classloader. "-" means that all classes under package "org.apache.sqoop.connector." should not be considered system class as we want these classes loaded by the connector's own classloader, not the system classloader.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > connector/connector-ftp/pom.xml, line 96
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131928#file1131928line96>
> >
> > Let's not hardcode the project name here - it will be hard to mantain and prone to errors. We should be able to get the project name from ${project.artifactId} or similar variable.
Good point. Fixed.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > connector/connector-ftp/pom.xml, line 80
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131928#file1131928line80>
> >
> > Can we define the assembly plugin in root pom.xml so that we don't have repeat the version every time in each children pom file? We do that alredy for pretty much all dependencies (there are few exceptions) and it's much cleaner.
Good point. Have addressed this in the new patch.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > connector/connector-ftp/pom.xml, lines 88-102
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131928#file1131928line88>
> >
> > Wondering if there is a way to somehow put this fragment to some of the parent pom.xml files so that we don't have to repeat it every time?
> >
> > Perhaps we can create new parent module "sqoop-connector" that will contain the packaging portion and use it for all children?
Good point. Addressed in the new patch.
> On Dec. 5, 2015, 12:35 p.m., Jarek Cecho wrote:
> > core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java, lines 184-200
> > <https://reviews.apache.org/r/39586/diff/9/?file=1131941#file1131941line184>
> >
> > I'm wondering if there is a way to avoid creating the intermediate files on disk?
> >
> > How are other similar libraries solving (One jar for example) the same problem?
Good point. Have filed JIRA SQOOP-2635 to handle this. It takes the similar approach to One Jar.
- Dian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review108543
-----------------------------------------------------------
On Nov. 19, 2015, 12:39 p.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2015, 12:39 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/main/java/org/apache/sqoop/utils/ConnectorClassLoader.java 4c42a78
> common/src/main/resources/org.apache.sqoop.connector-classloader.properties e2936a9
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> pom.xml 91721ce
> server/pom.xml 370a6a2
> shell/src/test/java/org/apache/sqoop/shell/TestCloneCommand.java 7a139ba
> shell/src/test/java/org/apache/sqoop/shell/TestCreateCommand.java 0316a60
> shell/src/test/java/org/apache/sqoop/shell/TestUpdateCommand.java d8f061a
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review108543
-----------------------------------------------------------
My apologies for jumping so late on this review. Few high level notes:
1) It seems that the patch no longer applies cleanly, so you will have to rebase it.
2) I know that we've changed the design quite a bit - would you mind updating the JIRA and the review board summary with what exactly we're trying to accomplish here? I don't think that the title "Sqoop2: Allow connectors to express jar dependencies" is still relevant. It's more about wrap connector's dependencies inside the connector's jar, right?
3) This is rather huge change - I'm wondering if there are independent pieces (like changes to ConnectorClassLoader) that can be submitted via independent JIRA? It will make the subsequent reviews easier.
And few more tactical:
common/src/main/resources/org.apache.sqoop.connector-classloader.properties (line 54)
<https://reviews.apache.org/r/39586/#comment167983>
Wondering what is the "-" at the begging of the line mean?
connector/connector-ftp/pom.xml (line 80)
<https://reviews.apache.org/r/39586/#comment167985>
Can we define the assembly plugin in root pom.xml so that we don't have repeat the version every time in each children pom file? We do that alredy for pretty much all dependencies (there are few exceptions) and it's much cleaner.
connector/connector-ftp/pom.xml (lines 88 - 102)
<https://reviews.apache.org/r/39586/#comment167986>
Wondering if there is a way to somehow put this fragment to some of the parent pom.xml files so that we don't have to repeat it every time?
Perhaps we can create new parent module "sqoop-connector" that will contain the packaging portion and use it for all children?
connector/connector-ftp/pom.xml (line 96)
<https://reviews.apache.org/r/39586/#comment167987>
Let's not hardcode the project name here - it will be hard to mantain and prone to errors. We should be able to get the project name from ${project.artifactId} or similar variable.
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java (lines 60 - 68)
<https://reviews.apache.org/r/39586/#comment167988>
Aren't those dependecies on inside the connector's jar as well?
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java (lines 74 - 76)
<https://reviews.apache.org/r/39586/#comment168502>
Can we abstract this "magic" to standalone method and provide proper test coverage?
It seems fragile, so having good tests will hopefully help ensuring that we won't break this in the future.
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java (line 54)
<https://reviews.apache.org/r/39586/#comment168503>
Wondering why do we need to keep the classloader dependencies around? Can't we just store the classloder instance inside the ConnectorHandler or perhaps even easier call connectorInstance.getClass().getClassLoader().
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java (lines 183 - 199)
<https://reviews.apache.org/r/39586/#comment168504>
I'm wondering if there is a way to avoid creating the intermediate files on disk?
How are other similar libraries solving (One jar for example) the same problem?
core/src/main/java/org/apache/sqoop/driver/JobManager.java (lines 490 - 494)
<https://reviews.apache.org/r/39586/#comment168505>
I'm wondering if this is needed? Based on preliminary research any "new" call inside the initializer() method should use the Connector classes classloader.
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java (line 86)
<https://reviews.apache.org/r/39586/#comment168506>
We're not really storing the classpath here, just the jar location. Also I'm wondering if this is really desirable to store since the path is absolute on given machine, right?
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java (lines 61 - 78)
<https://reviews.apache.org/r/39586/#comment168507>
I think that easier and more nicer way how to do this is to load the SqoopConnector class with given class loader. Everything else we should "get" for free.
server/pom.xml (lines 83 - 88)
<https://reviews.apache.org/r/39586/#comment168508>
I'm wondering why do we need the exclusions? They shouldn't be propagated because we marked them as "provided" anyway right?
- Jarek Cecho
On Nov. 19, 2015, 12:39 p.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2015, 12:39 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/main/java/org/apache/sqoop/utils/ConnectorClassLoader.java 4c42a78
> common/src/main/resources/org.apache.sqoop.connector-classloader.properties e2936a9
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> pom.xml 91721ce
> server/pom.xml 370a6a2
> shell/src/test/java/org/apache/sqoop/shell/TestCloneCommand.java 7a139ba
> shell/src/test/java/org/apache/sqoop/shell/TestCreateCommand.java 0316a60
> shell/src/test/java/org/apache/sqoop/shell/TestUpdateCommand.java d8f061a
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
> On Dec. 21, 2015, 12:58 p.m., Jarek Cecho wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java, lines 69-70
> > <https://reviews.apache.org/r/39586/diff/15/?file=1173680#file1173680line69>
> >
> > Further thinking: This property should store location of connector jar. But that location will differ on each node because server will have it somewhere else then individual mappers. It seems that the only thing we are doing with the value is to get the jar name (anything after final "/"), so I'm wondering if it would make sense to rename this to contain only connector name and use that?
> >
> > Or even more cleaner way would be to store from and to connector classname, then we can use the API getJarForClass to find the connectors jar name - this would be more in line with how we're shipping the rest of the classes. What do you think?
Good advice. I have updated the patch to use the connector name to find the connector jar name. The reason I didn't use the connector classname is that getJarForClass requires to create a Class object firstly and this will result to loading that class which isn't expected.
- Dian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review111457
-----------------------------------------------------------
On Dec. 22, 2015, 3:38 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Dec. 22, 2015, 3:38 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
> 1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
> 2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
> 3. Load connector class with the connector's own ConnectorClassLoader.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/main/java/org/apache/sqoop/error/code/MRExecutionError.java 21e5c82
> common/src/main/java/org/apache/sqoop/utils/ClassUtils.java ed68988
> common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
> common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
> core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 3dee8f6
> pom.xml 460273a
> server/pom.xml 370a6a2
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review111457
-----------------------------------------------------------
I've finished the review:
connector/connector-ftp/pom.xml (line 47)
<https://reviews.apache.org/r/39586/#comment171659>
Nit: Incorrect ident
core/src/main/java/org/apache/sqoop/core/SqoopClosure.java (line 24)
<https://reviews.apache.org/r/39586/#comment171661>
It seems that this interface doesn't provide much value over Java's build in Callable:
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Callable.html
Wondering if we should just reuse it then?
core/src/main/java/org/apache/sqoop/core/SqoopUtils.java (line 25)
<https://reviews.apache.org/r/39586/#comment171660>
Do you think that it would make sense to move this method to ClassUtils rather then creating new util class?
https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/utils/ClassUtils.java
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java (lines 69 - 70)
<https://reviews.apache.org/r/39586/#comment171662>
Further thinking: This property should store location of connector jar. But that location will differ on each node because server will have it somewhere else then individual mappers. It seems that the only thing we are doing with the value is to get the jar name (anything after final "/"), so I'm wondering if it would make sense to rename this to contain only connector name and use that?
Or even more cleaner way would be to store from and to connector classname, then we can use the API getJarForClass to find the connectors jar name - this would be more in line with how we're shipping the rest of the classes. What do you think?
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java (lines 203 - 205)
<https://reviews.apache.org/r/39586/#comment171663>
I'm wondering why we are using the tmpfiles rather then tmpjars? Having the connector jars on the classpath shouldn't affect anything right? I mean, even the server have the connector jars on it's classpath - we're just using different classloader to load classes from those jars.
- Jarek Cecho
On Dec. 21, 2015, 6:23 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2015, 6:23 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
> 1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
> 2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
> 3. Load connector class with the connector's own ConnectorClassLoader.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
> common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/core/SqoopClosure.java PRE-CREATION
> core/src/main/java/org/apache/sqoop/core/SqoopUtils.java PRE-CREATION
> core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
> core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 3dee8f6
> pom.xml 460273a
> server/pom.xml 370a6a2
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review111768
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Dec. 22, 2015, 5:04 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Dec. 22, 2015, 5:04 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
> 1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
> 2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
> 3. Load connector class with the connector's own ConnectorClassLoader.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/main/java/org/apache/sqoop/error/code/MRExecutionError.java 21e5c82
> common/src/main/java/org/apache/sqoop/utils/ClassUtils.java ed68988
> common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
> common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
> core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 3dee8f6
> pom.xml 460273a
> server/pom.xml 370a6a2
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 22, 2015, 5:04 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/error/code/MRExecutionError.java 21e5c82
common/src/main/java/org/apache/sqoop/utils/ClassUtils.java ed68988
common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 3dee8f6
pom.xml 460273a
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 22, 2015, 3:38 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/error/code/MRExecutionError.java 21e5c82
common/src/main/java/org/apache/sqoop/utils/ClassUtils.java ed68988
common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 3dee8f6
pom.xml 460273a
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 21, 2015, 6:23 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/core/SqoopClosure.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/SqoopUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java 3dee8f6
pom.xml 460273a
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 20, 2015, 5:34 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java ec48f82
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/core/SqoopClosure.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/SqoopUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml 460273a
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 18, 2015, 12:34 p.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/core/SqoopClosure.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/SqoopUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml 460273a
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
> On Dec. 18, 2015, 10:02 a.m., Jarek Cecho wrote:
> > I didn't fully finished the review, but I wanted to share at least partial feedback:
Hi Jarcec,
Thanks a lot for the review. Make sense to me. Updated the patch accordingly.
- Dian
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review111162
-----------------------------------------------------------
On Dec. 18, 2015, 12:34 p.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 12:34 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
> 1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
> 2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
> 3. Load connector class with the connector's own ConnectorClassLoader.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/classloader/ConnectorClassLoader.java 370de2a
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> common/src/test/java/org/apache/sqoop/classloader/TestConnectorClassLoader.java 87edf3d
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/core/SqoopClosure.java PRE-CREATION
> core/src/main/java/org/apache/sqoop/core/SqoopUtils.java PRE-CREATION
> core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
> core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> pom.xml 460273a
> server/pom.xml 370a6a2
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/#review111162
-----------------------------------------------------------
I didn't fully finished the review, but I wanted to share at least partial feedback:
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java (lines 111 - 126)
<https://reviews.apache.org/r/39586/#comment171186>
It seems that jarsForConnector will always be array of exactly one item, so I'm wondering if it make sense to have the list here rather then have simply array of size one?
core/src/main/java/org/apache/sqoop/driver/JobManager.java (lines 508 - 516)
<https://reviews.apache.org/r/39586/#comment171187>
I see this fragment on a lot of places, do you think that it would make sense to create a util class/method that will do the classloader magic?
Jarcec
- Jarek Cecho
On Dec. 17, 2015, 5:52 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39586/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 5:52 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2634
> https://issues.apache.org/jira/browse/SQOOP-2634
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
> 1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
> 2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
> 3. Load connector class with the connector's own ConnectorClassLoader.
>
>
> Diffs
> -----
>
> assemblies/pom.xml PRE-CREATION
> assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
> common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
> connector/connector-ftp/pom.xml 41ea026
> connector/connector-generic-jdbc/pom.xml 052f06d
> connector/connector-hdfs/pom.xml 022a024
> connector/connector-kafka/pom.xml e0f0684
> connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
> connector/connector-kite/pom.xml d8eaa8e
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
> connector/connector-oracle-jdbc/pom.xml 8186b3a
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
> connector/connector-sftp/pom.xml 312ac61
> connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
> core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
> core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
> core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
> core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
> core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
> execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
> pom.xml 460273a
> server/pom.xml 370a6a2
> submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
>
> Diff: https://reviews.apache.org/r/39586/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 17, 2015, 5:52 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f19f391
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java d3a750e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml 460273a
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 10, 2015, 9:55 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java 4f4b9d3
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml e66de4d
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 10, 2015, 9:50 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java 4f4b9d3
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml e66de4d
server/pom.xml 370a6a2
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Provide classpath isolation for
connectors and its dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Dec. 10, 2015, 9:45 a.m.)
Review request for Sqoop.
Summary (updated)
-----------------
Sqoop2: Provide classpath isolation for connectors and its dependencies
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description (updated)
-------
The aim of this JIRA is to provide classpath isolation for connectors and its dependencies. It's achieved in the following method:
1. Package connector jar with its dependencies. There will be a directory lib which contains the dependencies of the connector
2. Improve the ConnectorClassLoader to make it can load classes from the dependencies which are inside the connector jar (SQOOP-2635)
3. Load connector class with the connector's own ConnectorClassLoader.
Diffs
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ConnectorClassLoader.java 4c42a78
common/src/main/resources/org.apache.sqoop.connector-classloader.properties e2936a9
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml 91721ce
server/pom.xml 370a6a2
shell/src/test/java/org/apache/sqoop/shell/TestCloneCommand.java 7a139ba
shell/src/test/java/org/apache/sqoop/shell/TestCreateCommand.java 0316a60
shell/src/test/java/org/apache/sqoop/shell/TestUpdateCommand.java d8f061a
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Nov. 19, 2015, 12:39 p.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
assemblies/pom.xml PRE-CREATION
assemblies/src/main/resources/assemblies/sqoop-connector.xml PRE-CREATION
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ConnectorClassLoader.java 4c42a78
common/src/main/resources/org.apache.sqoop.connector-classloader.properties e2936a9
connector/connector-ftp/pom.xml 41ea026
connector/connector-generic-jdbc/pom.xml 052f06d
connector/connector-hdfs/pom.xml 022a024
connector/connector-kafka/pom.xml e0f0684
connector/connector-kafka/src/main/java/org/apache/sqoop/connector/kafka/KafkaToInitializer.java 923d1aa
connector/connector-kite/pom.xml d8eaa8e
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java 28c5bac
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java 50daba0
connector/connector-oracle-jdbc/pom.xml 8186b3a
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
connector/connector-sftp/pom.xml 312ac61
connector/connector-sftp/src/main/java/org/apache/sqoop/connector/sftp/SftpToInitializer.java bfb51ac
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3acd4a1
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRUtils.java PRE-CREATION
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 0623f7b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 7d20992
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 8c8526b
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 623d1f4
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 2463643
execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMatching.java d0b41d1
pom.xml 91721ce
server/pom.xml 370a6a2
shell/src/test/java/org/apache/sqoop/shell/TestCloneCommand.java 7a139ba
shell/src/test/java/org/apache/sqoop/shell/TestCreateCommand.java 0316a60
shell/src/test/java/org/apache/sqoop/shell/TestUpdateCommand.java d8f061a
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 30, 2015, 8:32 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ClassUtils.java ed68988
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f606950
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java a33bc05
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
core/src/main/java/org/apache/sqoop/driver/JobRequest.java cfa45b2
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 2895530
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java ee5e8d1
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 30, 2015, 8:26 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ClassUtils.java ed68988
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f606950
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java a33bc05
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
core/src/main/java/org/apache/sqoop/driver/JobRequest.java cfa45b2
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 2895530
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java ee5e8d1
shell/src/main/java/org/apache/sqoop/shell/utils/ConfigFiller.java 6a2a96d
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 30, 2015, 7:29 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java f606950
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 9f9be57
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java a33bc05
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/main/java/org/apache/sqoop/driver/JobManager.java 15ca796
core/src/main/java/org/apache/sqoop/driver/JobRequest.java cfa45b2
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java 423b3df
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 2895530
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java 737ceda
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java ee5e8d1
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 30, 2015, 5:41 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java 285e893
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 522ed08
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java f49fca3
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/main/java/org/apache/sqoop/driver/JobManager.java 0d230f9
core/src/main/java/org/apache/sqoop/driver/JobRequest.java cfa45b2
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java fd2d1b4
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 7c1ec18
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java df767e6
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 0302150
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 30, 2015, 5:16 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/error/code/ConnectorError.java 2f17d95
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java 285e893
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 522ed08
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java f49fca3
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/main/java/org/apache/sqoop/driver/JobManager.java 0d230f9
core/src/main/java/org/apache/sqoop/driver/JobRequest.java cfa45b2
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java fd2d1b4
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 7c1ec18
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java df767e6
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 0302150
shell/src/main/java/org/apache/sqoop/shell/utils/ConfigFiller.java 852330e
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 23, 2015, 10:07 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 85ba8be
core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1899bb7
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java 285e893
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 522ed08
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java f49fca3
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java fd2d1b4
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 7c1ec18
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java c8d210e
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java b7aa8c6
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 0302150
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39586: Sqoop2: Allow connectors to express jar
dependencies
Posted by Dian Fu <di...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39586/
-----------------------------------------------------------
(Updated Oct. 23, 2015, 9:37 a.m.)
Review request for Sqoop.
Bugs: SQOOP-2634
https://issues.apache.org/jira/browse/SQOOP-2634
Repository: sqoop-sqoop2
Description
-------
Currently Sqoop 2 has already provided the ability to config jar dependencies with property "org.apache.sqoop.classpath.extra". The limitation of this property is that we have to put all the dependencies together. It can't express jar dependencies for a specified connector. This capacity is useful as some connectors may have conflict jar dependencies. Put all the dependencies from different connectors together may cause problems.
Diffs (updated)
-----
common/src/main/java/org/apache/sqoop/utils/ContextUtils.java 25d6fcb
core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java 285e893
core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java 522ed08
core/src/main/java/org/apache/sqoop/connector/ConnectorUtils.java PRE-CREATION
core/src/main/java/org/apache/sqoop/core/ConfigurationConstants.java f49fca3
core/src/main/java/org/apache/sqoop/core/SqoopConfiguration.java 3c2f67e
core/src/test/java/org/apache/sqoop/connector/TestConnectorManagerUtils.java fd2d1b4
core/src/test/java/org/apache/sqoop/connector/TestConnectorUtils.java PRE-CREATION
dist/src/main/conf/sqoop.properties 7c1ec18
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java c8d210e
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java b7aa8c6
repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 0302150
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java f396783
Diff: https://reviews.apache.org/r/39586/diff/
Testing
-------
Thanks,
Dian Fu