You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Qian Xu <sx...@googlemail.com> on 2014/08/04 06:20:13 UTC
Review Request 24223: SQOOP-1390: Import data to HDFS as a set of Parquet
files
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
Review request for Sqoop.
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
pom-old.xml a8f4361
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/com/cloudera/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/com/cloudera/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java a5f72f7
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review50337
-----------------------------------------------------------
Now that you've added support for append, I want to warn you that there's a slight change to the behavior in the upcoming release. In 0.15.0, what you have works fine. In 0.16.0, we are changing the writeTo method so that it checks to see that the target is empty and we've added an "appendTo" alternative that does the append without that check. So when you update to a newer release, you'll have to make that change.
- Ryan Blue
On Aug. 11, 2014, 11:30 p.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2014, 11:30 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
> *Support both import from scratch and in append mode
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Included 4 test cases. All of them are executed successfully.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review50881
-----------------------------------------------------------
I do have couple of comments, but otherwise I feel that we are in a shape to commit this feature! Please do upload updated patch to both review board and JIRA. I can't commit a change that is not on JIRA due to licensing issues.
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java
<https://reviews.apache.org/r/24223/#comment88751>
Nit: Unused import.
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java
<https://reviews.apache.org/r/24223/#comment88756>
Since this is more a Avro specific record rather then Sqoop "Generic", might I suggest to rename this class to correspond with that? Something like "SqoopAvroGenericRecord" or "SqoopAvroRecord"?
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java
<https://reviews.apache.org/r/24223/#comment88758>
This seems as a copy&paste from AvroImportMapper class - let's perhaps put it to some AvroUtils and reuse the same code?
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
<https://reviews.apache.org/r/24223/#comment88750>
Nit: I believe that Hadoop prefers using NullWritable instead of Void.
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
<https://reviews.apache.org/r/24223/#comment88752>
I can see that we are using this pattern in other mappers as well, could you please create follow up JIRA to create the temporary directory inside task working dir rather then in tmp?
I think that we should fix that across the code base, so it doesn't seem right to do this in this particular JIRA :)
src/java/org/apache/sqoop/mapreduce/ParquetJob.java
<https://reviews.apache.org/r/24223/#comment88753>
Nit: Please don't use "asterisks" imports, we prefer iterating over all required imports.
src/java/org/apache/sqoop/orm/ClassWriter.java
<https://reviews.apache.org/r/24223/#comment88757>
I would rename this class in similar manner as the class SqoopGenericRecord itself.
Jarcec
- Jarek Cecho
On Aug. 14, 2014, 8:24 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 14, 2014, 8:24 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
> *Support both import from scratch and in append mode
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/java/org/apache/sqoop/util/AppendUtils.java 5eaaa95
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Included 4 test cases. All of them are executed successfully.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review50975
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Aug. 19, 2014, 4:31 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 19, 2014, 4:31 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
> *Support both import from scratch and in append mode
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/org/apache/sqoop/avro/AvroUtil.java PRE-CREATION
> src/java/org/apache/sqoop/lib/SqoopAvroRecord.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 289eb28
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/java/org/apache/sqoop/util/AppendUtils.java 5eaaa95
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Included 4 test cases. All of them are executed successfully.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 19, 2014, 12:31 p.m.)
Review request for Sqoop.
Changes
-------
Renamed SqoopGenericRecord to SqoopAvroRecord and created a helper class for Avro related common code.
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
*Support both import from scratch and in append mode
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs (updated)
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/org/apache/sqoop/avro/AvroUtil.java PRE-CREATION
src/java/org/apache/sqoop/lib/SqoopAvroRecord.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/AvroImportMapper.java 289eb28
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/java/org/apache/sqoop/util/AppendUtils.java 5eaaa95
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Included 4 test cases. All of them are executed successfully.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 14, 2014, 8:24 a.m.)
Review request for Sqoop.
Changes
-------
Did some code cleanup. The code has been fully tested in overwrite and append mode with diverse scenarios
Repository: sqoop-trunk
Description (updated)
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
*Support both import from scratch and in append mode
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs (updated)
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/java/org/apache/sqoop/util/AppendUtils.java 5eaaa95
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Included 4 test cases. All of them are executed successfully.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 12, 2014, 6:30 a.m.)
Review request for Sqoop.
Changes
-------
Support import in append mode as well and the description has been updated.
Repository: sqoop-trunk
Description (updated)
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
*Support both import from scratch and in append mode
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing (updated)
-------
Included 4 test cases. All of them are executed successfully.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 12, 2014, 6:13 a.m.)
Review request for Sqoop.
Changes
-------
1. support append mode
2. code cleanup
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs (updated)
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review50190
-----------------------------------------------------------
The only thing that still looks odd to me is that you have a subclass of DatasetKeyOutputFormat with no changes. Is that class necessary?
I think this is ready, although it is limited in a couple of ways that we may want to improve later:
* This doesn't have a way to define a partition strategy, which would be helpful for large datasets.
* This must create the dataset, rather than appending to an existing one.
I don't think these should hold up getting this committed, but I want to make you aware of them so that you can use those features of Kite later.
- Ryan Blue
On Aug. 10, 2014, 7:54 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2014, 7:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Manually tested with a MySQL database. Unit tests are being developed yet.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 10, 2014, 2:54 p.m.)
Review request for Sqoop.
Changes
-------
Now uses DatasetKeyOutputFormat without any modification. It should also very easy to be used for Avro files. In the patch, a SqoopGenericRecord is introduced, which implements GenericRecord. When compiling a job, the input class is now switched from SqoopRecord to SqoopGenericRecord.
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs (updated)
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/org/apache/sqoop/lib/SqoopGenericRecord.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/orm/ClassWriter.java 94ff576
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Ryan Blue <bl...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review49742
-----------------------------------------------------------
I agree with Joey, it would be better to use the DatasetKeyOutputFormat so you don't have to maintain one.
You might also consider implementing a wrapper for SqoopRecord that implements GenericRecord [1]. That would remove the need to copy the values from one map to the other.
[1]: http://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericRecord.html
- Ryan Blue
On Aug. 6, 2014, 12:56 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 6, 2014, 12:56 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Manually tested with a MySQL database. Unit tests are being developed yet.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Joey Echeverria <jo...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review49734
-----------------------------------------------------------
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java
<https://reviews.apache.org/r/24223/#comment87043>
Could you use org.kitesdk.data.mapreduce.DatasetKeyOutputFormat instead of writing your own?
- Joey Echeverria
On Aug. 6, 2014, 7:56 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 6, 2014, 7:56 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Manually tested with a MySQL database. Unit tests are being developed yet.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 6, 2014, 7:56 a.m.)
Review request for Sqoop.
Changes
-------
updates all problems upon last review
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs (updated)
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/#review49642
-----------------------------------------------------------
First pass... comments below!
ivy.xml
<https://reviews.apache.org/r/24223/#comment86918>
Do we need to include kitesdk for hadoop1 and hadoop2? See avro dependency for an example of how to do this if we do need to.
pom-old.xml
<https://reviews.apache.org/r/24223/#comment86916>
The dependencies can exist in ivy only. There's no need to include in this pom file.
pom-old.xml
<https://reviews.apache.org/r/24223/#comment86917>
Same as above.
src/java/com/cloudera/sqoop/mapreduce/ParquetImportMapper.java
<https://reviews.apache.org/r/24223/#comment86890>
com.cloudera.x is deprecated. No need to provide.
src/java/com/cloudera/sqoop/mapreduce/ParquetOutputFormat.java
<https://reviews.apache.org/r/24223/#comment86891>
com.cloudera.x is deprecated. No need to provide.
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java
<https://reviews.apache.org/r/24223/#comment86889>
You can get rid of this. The com.cloudera.x packages are not maintained any more.
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
<https://reviews.apache.org/r/24223/#comment86892>
This is a bit confusing... could you add a few comments as to why an Avro schema would be used with the ParquetJob?
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java
<https://reviews.apache.org/r/24223/#comment86898>
I don't believe this is possible. Perhaps you were looking for "Boolean"?
- Abraham Elmahrek
On Aug. 5, 2014, 6:25 a.m., Qian Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24223/
> -----------------------------------------------------------
>
> (Updated Aug. 5, 2014, 6:25 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
> Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
>
> The major items are listed as follows:
> *Implement `ParquetImportMapper`
> *Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
>
> As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
>
>
> Diffs
> -----
>
> ivy.xml abc12a1
> ivy/libraries.properties a59471e
> pom-old.xml a8f4361
> src/docs/man/import-args.txt a4ce4ec
> src/docs/man/sqoop-import-all-tables.txt 6b639f5
> src/docs/user/hcatalog.txt cd1dde3
> src/docs/user/help.txt a9e1e89
> src/docs/user/import-all-tables.txt 60645f1
> src/docs/user/import.txt 192e97e
> src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
> src/java/com/cloudera/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/com/cloudera/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
> src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java a5f72f7
> src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
> src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
> src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
> src/licenses/LICENSE-BIN.txt 4215d26
> src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/24223/diff/
>
>
> Testing
> -------
>
> Manually tested with a MySQL database. Unit tests are being developed yet.
>
>
> Thanks,
>
> Qian Xu
>
>
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 5, 2014, 6:25 a.m.)
Review request for Sqoop.
Changes
-------
provided an updated patch with unit tests
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs (updated)
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
pom-old.xml a8f4361
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/com/cloudera/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/com/cloudera/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java a5f72f7
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
src/test/com/cloudera/sqoop/TestParquetImport.java PRE-CREATION
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 5, 2014, 6:24 a.m.)
Review request for Sqoop.
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
pom-old.xml a8f4361
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/com/cloudera/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/com/cloudera/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java a5f72f7
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
Thanks,
Qian Xu
Re: Review Request 24223: SQOOP-1390: Import data to HDFS as a set of
Parquet files
Posted by Qian Xu <sx...@googlemail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24223/
-----------------------------------------------------------
(Updated Aug. 5, 2014, 6:19 a.m.)
Review request for Sqoop.
Changes
-------
patch with unit tests
Repository: sqoop-trunk
Description
-------
The patch proposes to add the possibility to import an individual table from a RDBMS into HDFS as a set of Parquet files. It also supports a command-line interface with a new argument `--as-parquetfile`
Example invocation: `sqoop import --connect JDBC_URI --table TABLE --as-parquetfile --target-dir /path/to/files`
The major items are listed as follows:
*Implement `ParquetImportMapper`
*Hook up the `ParquetOutputFormat` and `ParquetImportMapper` in the import job.
As Parquet is a columnar storage format, it doesn't make sense to write to it directly from record-based tools. We've considered of using Kite SDK to simplify the handling of Parquet specific things. The major idea is to convert `SqoopRecord` as `GenericRecord` and write them into a Kite dataset. Kite SDK will convert these records to as a set of Parquet files.
Diffs
-----
ivy.xml abc12a1
ivy/libraries.properties a59471e
pom-old.xml a8f4361
src/docs/man/import-args.txt a4ce4ec
src/docs/man/sqoop-import-all-tables.txt 6b639f5
src/docs/user/hcatalog.txt cd1dde3
src/docs/user/help.txt a9e1e89
src/docs/user/import-all-tables.txt 60645f1
src/docs/user/import.txt 192e97e
src/java/com/cloudera/sqoop/SqoopOptions.java ffec2dc
src/java/com/cloudera/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/com/cloudera/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java a5f72f7
src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 6dcfebb
src/java/org/apache/sqoop/mapreduce/ParquetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetJob.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/ParquetOutputFormat.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java b77b1ea
src/java/org/apache/sqoop/tool/ImportTool.java a3a2d0d
src/licenses/LICENSE-BIN.txt 4215d26
Diff: https://reviews.apache.org/r/24223/diff/
Testing
-------
Manually tested with a MySQL database. Unit tests are being developed yet.
File Attachments (updated)
----------------
patch with unit tests
https://reviews.apache.org/media/uploaded/files/2014/08/05/d28b05dd-c608-4cf0-89b2-35354b79ffb8__SQOOP-1390.patch
Thanks,
Qian Xu