You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Jarek Cecho <ja...@apache.org> on 2015/03/18 21:48:47 UTC

Review Request 32216: SQOOP-2228 Sqoop2: HDFS Connector: Import data to temporary directory before moving them to target directory

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32216/
-----------------------------------------------------------

Review request for Sqoop.


Repository: sqoop-sqoop2


Description
-------

Implemented as per description in JIRA.


Diffs
-----

  common/src/main/java/org/apache/sqoop/error/code/HdfsConnectorError.java c85e7fc 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java bd74bec 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 0ced6d0 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java 3c85be8 
  connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java 83bac27 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java b7c81ec 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToDestroyer.java PRE-CREATION 
  connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToInitializer.java 1daa25a 

Diff: https://reviews.apache.org/r/32216/diff/


Testing
-------

I've added unit tests for new functionality. Existing integration tests passes without any changes (which is a positive thing here).


Thanks,

Jarek Cecho


Re: Review Request 32216: SQOOP-2228 Sqoop2: HDFS Connector: Import data to temporary directory before moving them to target directory

Posted by Jarek Cecho <ja...@apache.org>.

> On March 18, 2015, 9:28 p.m., Abraham Elmahrek wrote:
> > Might be cool to make this configurable for HDFS encryption/ACLs?

I would prefer not to make it configurable but instead always write to a subdirectory of target directory because that will solve the issue with HDFS encryption/ACLs.

I'm open to do it if we feel that it will be required.


> On March 18, 2015, 9:28 p.m., Abraham Elmahrek wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java, lines 58-62
> > <https://reviews.apache.org/r/32216/diff/1/?file=899373#file899373line58>
> >
> >     Why not move entire directory? instead of file at a time.

Good question. There are several reasons why I've decided to go route of moving one file at the time:
* The working directory is subdirectory of the target directory (e.g. I can't move entire directory to it's parent)
* Also this is prepared for incremental import where I will need to append files to existing directory


> On March 18, 2015, 9:28 p.m., Abraham Elmahrek wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java, line 60
> > <https://reviews.apache.org/r/32216/diff/1/?file=899373#file899373line60>
> >
> >     Debug maybe.

This code runs on mapreduce side where it's not "that easy" to apply our log4j configuration to enable debug mode. I would prefer to keep it as info, so that it will always appear in the logs.


> On March 18, 2015, 9:28 p.m., Abraham Elmahrek wrote:
> > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java, lines 50-53
> > <https://reviews.apache.org/r/32216/diff/1/?file=899374#file899374line50>
> >
> >     Not opposed to it... but these probably can be removed since they're only useful for debugging

It's a good reminder of what conditions we're expecting to be true as they are checked elsewhere. I've started adding it to other places as well - I think that it will help when multiple developpers will look at the same file to realize that we're simply assuming those conditions without any additional checks.


- Jarek


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32216/#review76949
-----------------------------------------------------------


On March 18, 2015, 8:48 p.m., Jarek Cecho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32216/
> -----------------------------------------------------------
> 
> (Updated March 18, 2015, 8:48 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> Implemented as per description in JIRA.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/sqoop/error/code/HdfsConnectorError.java c85e7fc 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java bd74bec 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 0ced6d0 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java 3c85be8 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java 83bac27 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java b7c81ec 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToDestroyer.java PRE-CREATION 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToInitializer.java 1daa25a 
> 
> Diff: https://reviews.apache.org/r/32216/diff/
> 
> 
> Testing
> -------
> 
> I've added unit tests for new functionality. Existing integration tests passes without any changes (which is a positive thing here).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>


Re: Review Request 32216: SQOOP-2228 Sqoop2: HDFS Connector: Import data to temporary directory before moving them to target directory

Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32216/#review76949
-----------------------------------------------------------


Might be cool to make this configurable for HDFS encryption/ACLs?


connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
<https://reviews.apache.org/r/32216/#comment124684>

    Why not move entire directory? instead of file at a time.



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java
<https://reviews.apache.org/r/32216/#comment124681>

    Debug maybe.



connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java
<https://reviews.apache.org/r/32216/#comment124686>

    Not opposed to it... but these probably can be removed since they're only useful for debugging


- Abraham Elmahrek


On March 18, 2015, 8:48 p.m., Jarek Cecho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32216/
> -----------------------------------------------------------
> 
> (Updated March 18, 2015, 8:48 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> Implemented as per description in JIRA.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/sqoop/error/code/HdfsConnectorError.java c85e7fc 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java bd74bec 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 0ced6d0 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java 3c85be8 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java 83bac27 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java b7c81ec 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToDestroyer.java PRE-CREATION 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToInitializer.java 1daa25a 
> 
> Diff: https://reviews.apache.org/r/32216/diff/
> 
> 
> Testing
> -------
> 
> I've added unit tests for new functionality. Existing integration tests passes without any changes (which is a positive thing here).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>


Re: Review Request 32216: SQOOP-2228 Sqoop2: HDFS Connector: Import data to temporary directory before moving them to target directory

Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32216/#review76951
-----------------------------------------------------------

Ship it!


Ship It!

- Abraham Elmahrek


On March 18, 2015, 8:48 p.m., Jarek Cecho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32216/
> -----------------------------------------------------------
> 
> (Updated March 18, 2015, 8:48 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> Implemented as per description in JIRA.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/sqoop/error/code/HdfsConnectorError.java c85e7fc 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConstants.java bd74bec 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsLoader.java 0ced6d0 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToDestroyer.java 3c85be8 
>   connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsToInitializer.java 83bac27 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestLoader.java b7c81ec 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToDestroyer.java PRE-CREATION 
>   connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestToInitializer.java 1daa25a 
> 
> Diff: https://reviews.apache.org/r/32216/diff/
> 
> 
> Testing
> -------
> 
> I've added unit tests for new functionality. Existing integration tests passes without any changes (which is a positive thing here).
> 
> 
> Thanks,
> 
> Jarek Cecho
> 
>