You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Illya Yalovyy <ya...@amazon.com> on 2017/02/16 22:22:48 UTC
Review Request 56770: SQOOP-3136 Sqoop should work well with not
default file systems
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56770/
-----------------------------------------------------------
Review request for Sqoop, Jarek Cecho and Venkat Ranganathan.
Repository: sqoop-trunk
Description
-------
Currently Sqoop assumes default file system when it comes to IO operations. It makes it hard to use other FileSystem implementations as source or destination.
https://issues.apache.org/jira/browse/SQOOP-3136
Diffs
-----
src/java/com/cloudera/sqoop/io/LobReaderCache.java 3394296
src/java/org/apache/sqoop/hive/HiveImport.java 4828375
src/java/org/apache/sqoop/hive/TableDefWriter.java c9962e9
src/java/org/apache/sqoop/io/LobReaderCache.java bd75374
src/java/org/apache/sqoop/io/SplittingOutputStream.java 5f98192
src/java/org/apache/sqoop/lib/LargeObjectLoader.java 70c0f4e
src/java/org/apache/sqoop/manager/oracle/OraOopUtilities.java e81588c
src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java e08f997
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 260bc29
src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 27f84da
src/java/org/apache/sqoop/mapreduce/HBaseBulkImportJob.java b32cdd1
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 626119b
src/java/org/apache/sqoop/mapreduce/JdbcUpdateExportJob.java f911280
src/java/org/apache/sqoop/mapreduce/MergeJob.java 5b6c4df
src/java/org/apache/sqoop/tool/ImportTool.java 258ef79
src/java/org/apache/sqoop/util/FileSystemUtil.java PRE-CREATION
src/java/org/apache/sqoop/util/FileUploader.java 155cffc
src/test/org/apache/sqoop/util/TestFileSystemUtil.java PRE-CREATION
Diff: https://reviews.apache.org/r/56770/diff/
Testing
-------
** Build:
ant clean package
...
BUILD SUCCESSFUL
Total time: 51 seconds
** Test:
ant test
...
BUILD SUCCESSFUL
Total time: 7 minutes 21 seconds
* On Hadoop Cluster:
** original version of sqoop:
sqoop import --connect <JDBC URL> --table table1 --driver <JDBC DRIVER> --username root --password **** --delete-target-dir --target-dir s3a://some-bucket/tmp/sqoop
...
17/02/15 19:16:59 ERROR tool.ImportTool: Imported Failed: Wrong FS: s3a://some-bucket/tmp/sqoop, expected: hdfs://<DNS>:8020
** updated version of sqoop:
sqoop import --connect <JDBC URL> --table table1 --driver <JDBC DRIVER> --username root --password **** --delete-target-dir --target-dir s3a://some-bucket/tmp/sqoop
...
17/02/15 22:24:42 INFO mapreduce.Job: Running job: job_1487183144282_0004
17/02/15 22:24:52 INFO mapreduce.Job: Job job_1487183144282_0004 running in uber mode : false
17/02/15 22:24:52 INFO mapreduce.Job: map 0% reduce 0%
17/02/15 22:25:04 INFO mapreduce.Job: map 25% reduce 0%
17/02/15 22:25:06 INFO mapreduce.Job: map 50% reduce 0%
17/02/15 22:25:07 INFO mapreduce.Job: map 75% reduce 0%
17/02/15 22:25:08 INFO mapreduce.Job: map 100% reduce 0%
17/02/15 22:25:08 INFO mapreduce.Job: Job job_1487183144282_0004 completed successfully
17/02/15 22:25:08 INFO mapreduce.Job: Counters: 36
...
17/02/15 22:25:08 INFO mapreduce.ImportJobBase: Retrieved 4993 records.
Thanks,
Illya Yalovyy