You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Dian Fu <di...@gmail.com> on 2015/10/23 08:24:42 UTC
Review Request 39581: Sqoop2: add partition class to FROM
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39581/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-2632
https://issues.apache.org/jira/browse/SQOOP-2632
Repository: sqoop-sqoop2
Description
-------
Each SqoopMapper will process one SqoopSplit. SqoopSplit is instantiated by the MR framework. Let's suppose that MR framework loads SqoopSplit with classloader A. As SqoopSplit has one field partition, so when SqoopSplit is loaded by classloader A, class Partition will also be loaded by classloader A and field partition will be of type A.Partition.
In SqoopMapper, we will call extractor.extract(extractorContext, fromConfig, fromJob, partition). To provide classpath isolation, we will load class Extractor with a custom classloader B, so class Partition in Extractor will be loaded by classloader B. This will cause type miss-match between A.Partition and B.Partition when we call extractor.extract(extractorContext, fromConfig, fromJob, partition.
To solve this issue, we need to make sure class Partition not be loaded by the custom classloader B. To achieve this, we need firstly provide a mechanism to let SqoopMapper know the class name of Partition.
Diffs
-----
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 113465a
connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnector.java 7e7c022
connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java ca860b1
connector/connector-sdk/src/main/java/org/apache/sqoop/job/etl/From.java e9d2abe
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java c8d210e
execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java b7aa8c6
Diff: https://reviews.apache.org/r/39581/diff/
Testing
-------
Thanks,
Dian Fu
Re: Review Request 39581: Sqoop2: add partition class to FROM
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39581/#review103845
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Oct. 23, 2015, 6:24 a.m., Dian Fu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39581/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2015, 6:24 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2632
> https://issues.apache.org/jira/browse/SQOOP-2632
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Each SqoopMapper will process one SqoopSplit. SqoopSplit is instantiated by the MR framework. Let's suppose that MR framework loads SqoopSplit with classloader A. As SqoopSplit has one field partition, so when SqoopSplit is loaded by classloader A, class Partition will also be loaded by classloader A and field partition will be of type A.Partition.
> In SqoopMapper, we will call extractor.extract(extractorContext, fromConfig, fromJob, partition). To provide classpath isolation, we will load class Extractor with a custom classloader B, so class Partition in Extractor will be loaded by classloader B. This will cause type miss-match between A.Partition and B.Partition when we call extractor.extract(extractorContext, fromConfig, fromJob, partition.
> To solve this issue, we need to make sure class Partition not be loaded by the custom classloader B. To achieve this, we need firstly provide a mechanism to let SqoopMapper know the class name of Partition.
>
>
> Diffs
> -----
>
> connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 113465a
> connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnector.java 7e7c022
> connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteConnector.java ca860b1
> connector/connector-sdk/src/main/java/org/apache/sqoop/job/etl/From.java e9d2abe
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java c8d210e
> execution/mapreduce/src/main/java/org/apache/sqoop/job/MRJobConstants.java b7aa8c6
>
> Diff: https://reviews.apache.org/r/39581/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Dian Fu
>
>