You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Mariappan Asokan <ma...@syncsort.com> on 2014/06/15 00:47:01 UTC
Review Request 22516: Support importing mainframe sequential datasets
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/
-----------------------------------------------------------
Review request for Sqoop.
Repository: sqoop-trunk
Description
-------
This is to move mainframe datasets to Hadoop.
Diffs
-----
src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
Diff: https://reviews.apache.org/r/22516/diff/
Testing
-------
Thanks,
Mariappan Asokan
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Jarek Cecho <ja...@apache.org>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
>
> Venkat Ranganathan wrote:
> Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
>
> Mariappan Asokan wrote:
> Thanks for your suggestions. Sorry, I did not get back sooner. In Sqoop 1.x, there is a strong assumption that input source is always a database table. Due to this the sqoop import tool has many options that are relevant to a source database table. A mainframe source is totally different from a database table. I think it is better to create a separate tool for mainframe import rather than just a new connection manager. The mainframe import tool will not support many options that the database import tool supports. It will have its own options that the database import tool does not support. At present, these are the host name and partitioned dataset name. In the future, the mainframe import tool may be enhanced with metadata specific or connection specific arguments unique to mainframe. Creating a synthetic URI for a connection seems to be somewhat artificial to me.
>
> Contrary to what I stated before, considering possible future enhancements, I think it is better to retain the support for parsing input format, output format, Hive, HBase, HCatalog, and codegen options. The documentation will be enhanced in the future to reflect this support.
>
>
> Venkat Ranganathan wrote:
> Thanks for your thoughts on the suggestion. As you correctly pointed out, Sqoop 1.x has a JDBC model (that is why you had to implement a ConnectionManager and provide pseudo values for column types etc (always returning VARCHAR). I understand there will be options mainframe import will not support (much like there are mysql specific options or netezza or sqlserver specific options). I understand you want to have specific metadata for mainframe import. That may be tricky. Connection specific arguments can be implemented as how JDBC connection specific arguments are done.
>
> The reason for my suggestion was primarily to piggy back on the implementation for imports into hive/hbase in future when you have the ability to provide specific metadata on the data.
> You can definitely parse the various options, but you have to explicitly check and exit if the unsupported options are currently used.
>
> My only worry with this tool is that this may be one off for mainframe imports alone and we will be starting off with hdfs import only until you get to the rest of the parts and when we finally see this, it is basically duplicating some of the code and may be difficult to maintain,
>
>
> Gwen Shapira wrote:
> I just checked the possibility of adding non-JDBC imports as part of the import tool, using fake connection URL as you suggested.
> This is not feasible - ConnManager (which you need to inherit) has to implement getConnection, which returns java.sql.Connection. You can't return this connection object for an FTP. Same for readTable which must return a ResultSet.
>
> I think a separate tool is the only way to go.
>
> Gwen Shapira wrote:
> Never mind :)
> I missed the fact that the Mainframe tool actually extends ConnManager anyways.
>
> Mariappan Asokan wrote:
> Thanks for all your comments. I have listed the pros and cons of the a separate mainframe import tool. I would like to get the opinions of Sqoop committers and go with the decision of majority. If the decision is "no new import tool", I will make necessary changes in the code and documentation and upload a new patch.
>
> Pros:
>
> Mainframe source is entirely different from a database table. Several of the database related options (--boundary-query, --columns, --direct, --fetch-size, --inline-lob-limit, --null-string, null-non-string, --query, --split-by, and --table) are not meaningful and will not be supported. It is easier for users to understand the documentation. In the implementation, the
> options are validated syntactically rather than semantically. The mainframe host name can be specified as an argument to the --connect options. There is no synthetic JDBC type URI. Enhanced implementations of MainframeConnectionManager that support mainframe record layout or special connection methods can add extra arguments after "--".
>
> Con:
>
> There is some code duplication in processing the options for import targets.
>
I would personally prefer the cleaner approach with creating a new tool for the mainframe integration. I do agree with Venkat that it will require some code duplication and will be harder to maintain. On the other hand it seems to me that we can accomplish much better usability with the standalone tool (different set of parameters, our own validation rules, ...) which I would prefer slightly more.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Venkat Ranganathan <n....@live.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
>
> Venkat Ranganathan wrote:
> Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
>
> Mariappan Asokan wrote:
> Thanks for your suggestions. Sorry, I did not get back sooner. In Sqoop 1.x, there is a strong assumption that input source is always a database table. Due to this the sqoop import tool has many options that are relevant to a source database table. A mainframe source is totally different from a database table. I think it is better to create a separate tool for mainframe import rather than just a new connection manager. The mainframe import tool will not support many options that the database import tool supports. It will have its own options that the database import tool does not support. At present, these are the host name and partitioned dataset name. In the future, the mainframe import tool may be enhanced with metadata specific or connection specific arguments unique to mainframe. Creating a synthetic URI for a connection seems to be somewhat artificial to me.
>
> Contrary to what I stated before, considering possible future enhancements, I think it is better to retain the support for parsing input format, output format, Hive, HBase, HCatalog, and codegen options. The documentation will be enhanced in the future to reflect this support.
>
Thanks for your thoughts on the suggestion. As you correctly pointed out, Sqoop 1.x has a JDBC model (that is why you had to implement a ConnectionManager and provide pseudo values for column types etc (always returning VARCHAR). I understand there will be options mainframe import will not support (much like there are mysql specific options or netezza or sqlserver specific options). I understand you want to have specific metadata for mainframe import. That may be tricky. Connection specific arguments can be implemented as how JDBC connection specific arguments are done.
The reason for my suggestion was primarily to piggy back on the implementation for imports into hive/hbase in future when you have the ability to provide specific metadata on the data.
You can definitely parse the various options, but you have to explicitly check and exit if the unsupported options are currently used.
My only worry with this tool is that this may be one off for mainframe imports alone and we will be starting off with hdfs import only until you get to the rest of the parts and when we finally see this, it is basically duplicating some of the code and may be difficult to maintain,
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > Thanks for working on this and sorry for being late. I do have some review comments. Please see below.
Thanks for the review.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 62
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line62>
> >
> > Very Minor nits: We have a few lines with trailing blanks. IT would be great to fix them so that we can reduce more checkstyle errors creeping in. ant checkstyle target can be used to identify the issues reported and fix
I will fix these minor nits in the next version of the patch.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 122
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line122>
> >
> > Minor nit: Spaces between type and array declaration will be flagged by checkstyle also
I will fix this.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 85
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line85>
> >
> > Same question as HBaseTable
Please see my answer above.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Venkat Ranganathan <n....@live.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Gwen Shapira <gs...@cloudera.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
>
> Venkat Ranganathan wrote:
> Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
>
> Mariappan Asokan wrote:
> Thanks for your suggestions. Sorry, I did not get back sooner. In Sqoop 1.x, there is a strong assumption that input source is always a database table. Due to this the sqoop import tool has many options that are relevant to a source database table. A mainframe source is totally different from a database table. I think it is better to create a separate tool for mainframe import rather than just a new connection manager. The mainframe import tool will not support many options that the database import tool supports. It will have its own options that the database import tool does not support. At present, these are the host name and partitioned dataset name. In the future, the mainframe import tool may be enhanced with metadata specific or connection specific arguments unique to mainframe. Creating a synthetic URI for a connection seems to be somewhat artificial to me.
>
> Contrary to what I stated before, considering possible future enhancements, I think it is better to retain the support for parsing input format, output format, Hive, HBase, HCatalog, and codegen options. The documentation will be enhanced in the future to reflect this support.
>
>
> Venkat Ranganathan wrote:
> Thanks for your thoughts on the suggestion. As you correctly pointed out, Sqoop 1.x has a JDBC model (that is why you had to implement a ConnectionManager and provide pseudo values for column types etc (always returning VARCHAR). I understand there will be options mainframe import will not support (much like there are mysql specific options or netezza or sqlserver specific options). I understand you want to have specific metadata for mainframe import. That may be tricky. Connection specific arguments can be implemented as how JDBC connection specific arguments are done.
>
> The reason for my suggestion was primarily to piggy back on the implementation for imports into hive/hbase in future when you have the ability to provide specific metadata on the data.
> You can definitely parse the various options, but you have to explicitly check and exit if the unsupported options are currently used.
>
> My only worry with this tool is that this may be one off for mainframe imports alone and we will be starting off with hdfs import only until you get to the rest of the parts and when we finally see this, it is basically duplicating some of the code and may be difficult to maintain,
>
I just checked the possibility of adding non-JDBC imports as part of the import tool, using fake connection URL as you suggested.
This is not feasible - ConnManager (which you need to inherit) has to implement getConnection, which returns java.sql.Connection. You can't return this connection object for an FTP. Same for readTable which must return a ResultSet.
I think a separate tool is the only way to go.
- Gwen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Gwen Shapira <gs...@cloudera.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
>
> Venkat Ranganathan wrote:
> Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
>
> Mariappan Asokan wrote:
> Thanks for your suggestions. Sorry, I did not get back sooner. In Sqoop 1.x, there is a strong assumption that input source is always a database table. Due to this the sqoop import tool has many options that are relevant to a source database table. A mainframe source is totally different from a database table. I think it is better to create a separate tool for mainframe import rather than just a new connection manager. The mainframe import tool will not support many options that the database import tool supports. It will have its own options that the database import tool does not support. At present, these are the host name and partitioned dataset name. In the future, the mainframe import tool may be enhanced with metadata specific or connection specific arguments unique to mainframe. Creating a synthetic URI for a connection seems to be somewhat artificial to me.
>
> Contrary to what I stated before, considering possible future enhancements, I think it is better to retain the support for parsing input format, output format, Hive, HBase, HCatalog, and codegen options. The documentation will be enhanced in the future to reflect this support.
>
>
> Venkat Ranganathan wrote:
> Thanks for your thoughts on the suggestion. As you correctly pointed out, Sqoop 1.x has a JDBC model (that is why you had to implement a ConnectionManager and provide pseudo values for column types etc (always returning VARCHAR). I understand there will be options mainframe import will not support (much like there are mysql specific options or netezza or sqlserver specific options). I understand you want to have specific metadata for mainframe import. That may be tricky. Connection specific arguments can be implemented as how JDBC connection specific arguments are done.
>
> The reason for my suggestion was primarily to piggy back on the implementation for imports into hive/hbase in future when you have the ability to provide specific metadata on the data.
> You can definitely parse the various options, but you have to explicitly check and exit if the unsupported options are currently used.
>
> My only worry with this tool is that this may be one off for mainframe imports alone and we will be starting off with hdfs import only until you get to the rest of the parts and when we finally see this, it is basically duplicating some of the code and may be difficult to maintain,
>
>
> Gwen Shapira wrote:
> I just checked the possibility of adding non-JDBC imports as part of the import tool, using fake connection URL as you suggested.
> This is not feasible - ConnManager (which you need to inherit) has to implement getConnection, which returns java.sql.Connection. You can't return this connection object for an FTP. Same for readTable which must return a ResultSet.
>
> I think a separate tool is the only way to go.
Never mind :)
I missed the fact that the Mainframe tool actually extends ConnManager anyways.
- Gwen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Venkat Ranganathan <n....@live.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 62
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line62>
> >
> > Very Minor nits: We have a few lines with trailing blanks. IT would be great to fix them so that we can reduce more checkstyle errors creeping in. ant checkstyle target can be used to identify the issues reported and fix
>
> Mariappan Asokan wrote:
> I will fix these minor nits in the next version of the patch.
Thanks
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 85
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line85>
> >
> > Same question as HBaseTable
>
> Mariappan Asokan wrote:
> Please see my answer above.
Thanks
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 122
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line122>
> >
> > Minor nit: Spaces between type and array declaration will be flagged by checkstyle also
>
> Mariappan Asokan wrote:
> I will fix this.
Thanks
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
>
> Venkat Ranganathan wrote:
> Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
>
> Mariappan Asokan wrote:
> Thanks for your suggestions. Sorry, I did not get back sooner. In Sqoop 1.x, there is a strong assumption that input source is always a database table. Due to this the sqoop import tool has many options that are relevant to a source database table. A mainframe source is totally different from a database table. I think it is better to create a separate tool for mainframe import rather than just a new connection manager. The mainframe import tool will not support many options that the database import tool supports. It will have its own options that the database import tool does not support. At present, these are the host name and partitioned dataset name. In the future, the mainframe import tool may be enhanced with metadata specific or connection specific arguments unique to mainframe. Creating a synthetic URI for a connection seems to be somewhat artificial to me.
>
> Contrary to what I stated before, considering possible future enhancements, I think it is better to retain the support for parsing input format, output format, Hive, HBase, HCatalog, and codegen options. The documentation will be enhanced in the future to reflect this support.
>
>
> Venkat Ranganathan wrote:
> Thanks for your thoughts on the suggestion. As you correctly pointed out, Sqoop 1.x has a JDBC model (that is why you had to implement a ConnectionManager and provide pseudo values for column types etc (always returning VARCHAR). I understand there will be options mainframe import will not support (much like there are mysql specific options or netezza or sqlserver specific options). I understand you want to have specific metadata for mainframe import. That may be tricky. Connection specific arguments can be implemented as how JDBC connection specific arguments are done.
>
> The reason for my suggestion was primarily to piggy back on the implementation for imports into hive/hbase in future when you have the ability to provide specific metadata on the data.
> You can definitely parse the various options, but you have to explicitly check and exit if the unsupported options are currently used.
>
> My only worry with this tool is that this may be one off for mainframe imports alone and we will be starting off with hdfs import only until you get to the rest of the parts and when we finally see this, it is basically duplicating some of the code and may be difficult to maintain,
>
>
> Gwen Shapira wrote:
> I just checked the possibility of adding non-JDBC imports as part of the import tool, using fake connection URL as you suggested.
> This is not feasible - ConnManager (which you need to inherit) has to implement getConnection, which returns java.sql.Connection. You can't return this connection object for an FTP. Same for readTable which must return a ResultSet.
>
> I think a separate tool is the only way to go.
>
> Gwen Shapira wrote:
> Never mind :)
> I missed the fact that the Mainframe tool actually extends ConnManager anyways.
Thanks for all your comments. I have listed the pros and cons of the a separate mainframe import tool. I would like to get the opinions of Sqoop committers and go with the decision of majority. If the decision is "no new import tool", I will make necessary changes in the code and documentation and upload a new patch.
Pros:
Mainframe source is entirely different from a database table. Several of the database related options (--boundary-query, --columns, --direct, --fetch-size, --inline-lob-limit, --null-string, null-non-string, --query, --split-by, and --table) are not meaningful and will not be supported. It is easier for users to understand the documentation. In the implementation, the
options are validated syntactically rather than semantically. The mainframe host name can be specified as an argument to the --connect options. There is no synthetic JDBC type URI. Enhanced implementations of MainframeConnectionManager that support mainframe record layout or special connection methods can add extra arguments after "--".
Con:
There is some code duplication in processing the options for import targets.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 10, 2014, 8:22 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 75
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line75>
> >
> > Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
>
> Mariappan Asokan wrote:
> Each record in a mainframe dataset is treated as a single field (or column.) So, theoretically HBase, Accumulo, and Hive are supported but with limited usability. So, I did not add them in the documentation. If you feel strongly that they should be documented, I can work on that in the next version of the patch.
>
> Venkat Ranganathan wrote:
> I feel it would be good to say we import only as text files and leave further processing, loading into hive/hbase upto the user as the composition of the records and needed processing differ and the schema can't be inferred.
>
> Mariappan Asokan wrote:
> I agree with you. To avoid confusion, I plan to remove support for parsing input format, output format, hive, hbase, hcatalog, and codegen options. This will synchronize the document with the code. What do you think?
>
>
> Venkat Ranganathan wrote:
> Sorry for the delay. I was wondering whether the mainframe connector can just define connector specific extra args and not create another tool. Please see NetezzaManager or DirectNetezzaManager as an example. May be you have to invent a new synthetic URI format say jdbc:mfftp:<host address>:<port>/dataset and choose your Connection Manager when --connect option with the above uri format is given. That should simplify a whole lot in my opinion. What do you think?
Thanks for your suggestions. Sorry, I did not get back sooner. In Sqoop 1.x, there is a strong assumption that input source is always a database table. Due to this the sqoop import tool has many options that are relevant to a source database table. A mainframe source is totally different from a database table. I think it is better to create a separate tool for mainframe import rather than just a new connection manager. The mainframe import tool will not support many options that the database import tool supports. It will have its own options that the database import tool does not support. At present, these are the host name and partitioned dataset name. In the future, the mainframe import tool may be enhanced with metadata specific or connection specific arguments unique to mainframe. Creating a synthetic URI for a connection seems to be somewhat artificial to me.
Contrary to what I stated before, considering possible future enhancements, I think it is better to retain the support for parsing input format, output format, Hive, HBase, HCatalog, and codegen options. The documentation will be enhanced in the future to reflect this support.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47555
-----------------------------------------------------------
Thanks for working on this and sorry for being late. I do have some review comments. Please see below.
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment83563>
Very Minor nits: We have a few lines with trailing blanks. IT would be great to fix them so that we can reduce more checkstyle errors creeping in. ant checkstyle target can be used to identify the issues reported and fix
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment83565>
Is import into Hbase and Accumulo supported by this tool? It looks like the only target supported is HDFS text files from the command help.
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment83566>
Same question as HBaseTable
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment83564>
Minor nit: Spaces between type and array declaration will be flagged by checkstyle also
- Venkat Ranganathan
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 10, 2014, 4:54 p.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 47
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line47>
> >
> > I would also suggest that you override isORMFacilitySelfManaged() and return true so that the ORM java class is not generated for this job.
Sure, I will do that.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47576
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 10, 2014, 4:54 p.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 103
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line103>
> >
> > Minor nit. This will be flagged by checkstyle. type[] should be used for arrays.
Fixed this in the latest patch.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47576
-----------------------------------------------------------
On Sept. 8, 2014, 11:19 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2014, 11:19 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-independent-args.txt PRE-CREATION
> src/docs/man/hbase-args.txt 456bc14
> src/docs/man/hive-args.txt dd77c3e
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47576
-----------------------------------------------------------
Couple more things I noticed
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment83632>
I would also suggest that you override isORMFacilitySelfManaged() and return true so that the ORM java class is not generated for this job.
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment83634>
Minor nit. This will be flagged by checkstyle. type[] should be used for arrays.
- Venkat Ranganathan
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On July 11, 2014, 6:20 p.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/tool/MainframeImportTool.java, line 1
> > <https://reviews.apache.org/r/22516/diff/1/?file=608155#file608155line1>
> >
> > I don't think a new tool is necessary. As I mentioned in my earlier comment, we can just define connector specific extra args as needed and use a synthetic URI to choose your connection manager.
Please see my reply above.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47665
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review47665
-----------------------------------------------------------
src/java/org/apache/sqoop/tool/MainframeImportTool.java
<https://reviews.apache.org/r/22516/#comment83786>
I don't think a new tool is necessary. As I mentioned in my earlier comment, we can just define connector specific extra args as needed and use a synthetic URI to choose your connection manager.
- Venkat Ranganathan
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > Hi Asokan,
> > thank you very much for taking up this huge change! I do have couple of high level comments:
> >
> > 1) It seems that you have several "trailing whitespaces" in the patch. Could you please clean them up? They will show up with a red mark here on review board, so they are easy to spot.
> >
> > 2) It seems that there are no docs - could you please add small paragraph to our user guide documenting the behaviour?
I cleaned up the spaces in the most recent version of the patch I posted in the Jira. Please refer to the latest patch for all documentation changes and provide feedback.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, line 50
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line50>
> >
> > Nit: Shoudn't this be private or protected constant?
This is referred by a test to verify correctness. I am leaving it as public.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/manager/MainframeManager.java, lines 168-170
> > <https://reviews.apache.org/r/22516/diff/1/?file=608148#file608148line168>
> >
> > I'm assuming that this comment should be deleted?
Correct.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java, lines 56-57
> > <https://reviews.apache.org/r/22516/diff/1/?file=608149#file608149line56>
> >
> > I don't particulary like the fact that we're reusing existing constant for table name for something else. What about creating a new constant for the mainframe dataset name?
I created a MainframeConfiguration class and defined the constant there. If it is deemed as an overkill, I can keep the new constant in DBConfiguration itself. Please provide feedback.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java, lines 71-72
> > <https://reviews.apache.org/r/22516/diff/1/?file=608150#file608150line71>
> >
> > Please use constants from ConfigurationConstants class.
Done in the recent patch.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java, lines 22-25
> > <https://reviews.apache.org/r/22516/diff/1/?file=608159#file608159line22>
> >
> > We should add mockito as dependency to ivy as well if we want to use this library.
Added in the recent patch.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review51116
-----------------------------------------------------------
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Jarek Cecho <ja...@apache.org>.
> On Aug. 25, 2014, 10 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java, lines 56-57
> > <https://reviews.apache.org/r/22516/diff/1/?file=608149#file608149line56>
> >
> > I don't particulary like the fact that we're reusing existing constant for table name for something else. What about creating a new constant for the mainframe dataset name?
>
> Mariappan Asokan wrote:
> I created a MainframeConfiguration class and defined the constant there. If it is deemed as an overkill, I can keep the new constant in DBConfiguration itself. Please provide feedback.
Having a special class seems completely reasonable.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review51116
-----------------------------------------------------------
On Aug. 27, 2014, 7:55 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2014, 7:55 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-connection-args.txt PRE-CREATION
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop-import.txt 00b1ec8
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review51116
-----------------------------------------------------------
Hi Asokan,
thank you very much for taking up this huge change! I do have couple of high level comments:
1) It seems that you have several "trailing whitespaces" in the patch. Could you please clean them up? They will show up with a red mark here on review board, so they are easy to spot.
2) It seems that there are no docs - could you please add small paragraph to our user guide documenting the behaviour?
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment89502>
Nit: Shoudn't this be private or protected constant?
src/java/org/apache/sqoop/manager/MainframeManager.java
<https://reviews.apache.org/r/22516/#comment89141>
I'm assuming that this comment should be deleted?
src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java
<https://reviews.apache.org/r/22516/#comment89503>
I don't particulary like the fact that we're reusing existing constant for table name for something else. What about creating a new constant for the mainframe dataset name?
src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java
<https://reviews.apache.org/r/22516/#comment89678>
Please use constants from ConfigurationConstants class.
src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java
<https://reviews.apache.org/r/22516/#comment89501>
We should add mockito as dependency to ivy as well if we want to use this library.
Jarcec
- Jarek Cecho
On June 14, 2014, 10:46 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated June 14, 2014, 10:46 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On Aug. 28, 2014, 12:31 a.m., Venkat Ranganathan wrote:
> >
Venkat, thanks for doing a thorough review. I am sorry, I could not get back with my reply sooner. I have to do extensive documentation updates. They are still not perfect due to inherent assumption everywhere that input comes from only a DB table.
I have uploaded the latest patch file that has all changes not just incremental changes from previous version of the patch. Please let me know whether that is okay.
> On Aug. 28, 2014, 12:31 a.m., Venkat Ranganathan wrote:
> > src/docs/man/common-args.txt, line 24
> > <https://reviews.apache.org/r/22516/diff/2/?file=670447#file670447line24>
> >
> > I see - you want to remove DB specific args from this and only use it for import specifc. Don't we need this option in other Sqoop tools (like export, eval, list-databases etc
I have fixed this by creating database-independent-args.txt which describes arguments that are independent of database input. For other tools, the DB specific arguments will be exposed in the documentation.
> On Aug. 28, 2014, 12:31 a.m., Venkat Ranganathan wrote:
> > src/docs/man/database-connection-args.txt, line 10
> > <https://reviews.apache.org/r/22516/diff/2/?file=670448#file670448line10>
> >
> > Trailing space
Removed.
> On Aug. 28, 2014, 12:31 a.m., Venkat Ranganathan wrote:
> > src/docs/man/import-args.txt, line 35
> > <https://reviews.apache.org/r/22516/diff/2/?file=670449#file670449line35>
> >
> > Intended?
I modified this since mainframe import tool will support importing to anything that sqoop-import supports.
> On Aug. 28, 2014, 12:31 a.m., Venkat Ranganathan wrote:
> > src/docs/man/import-args.txt, line 56
> > <https://reviews.apache.org/r/22516/diff/2/?file=670449#file670449line56>
> >
> > I think a few of the options have been removed. Are you moving it into another guide.
I have not removed any args. Please look at the latest version.
> On Aug. 28, 2014, 12:31 a.m., Venkat Ranganathan wrote:
> > src/docs/user/import.txt, line 212
> > <https://reviews.apache.org/r/22516/diff/2/?file=670461#file670461line212>
> >
> > Not sure why we need to move this into a new file
Chunk of documentation on distributed cache is common and so I moved to a separate file.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review51725
-----------------------------------------------------------
On Aug. 27, 2014, 7:55 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2014, 7:55 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-connection-args.txt PRE-CREATION
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop-import.txt 00b1ec8
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review51725
-----------------------------------------------------------
src/docs/man/common-args.txt
<https://reviews.apache.org/r/22516/#comment90277>
I see - you want to remove DB specific args from this and only use it for import specifc. Don't we need this option in other Sqoop tools (like export, eval, list-databases etc
src/docs/man/database-connection-args.txt
<https://reviews.apache.org/r/22516/#comment90278>
Trailing space
src/docs/man/import-args.txt
<https://reviews.apache.org/r/22516/#comment90279>
Intended?
src/docs/man/import-args.txt
<https://reviews.apache.org/r/22516/#comment90280>
I think a few of the options have been removed. Are you moving it into another guide.
src/docs/user/import.txt
<https://reviews.apache.org/r/22516/#comment90281>
Not sure why we need to move this into a new file
Thanks for doing a significant set of revisions and tests and document update since I reviewed lsat. I just did a quick review of the docs. I will go through the rest of the code and update later. As I explained during the phone call a few weeks back, a separate tool for a specialized ftp import seems an overkill with some code repetition, but having a separate tool does have its benefits. So, I am +0 on a dedicated tool for this.
- Venkat Ranganathan
On Aug. 27, 2014, 7:55 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2014, 7:55 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-connection-args.txt PRE-CREATION
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop-import.txt 00b1ec8
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Jarek Cecho <ja...@apache.org>.
> On Sept. 3, 2014, 5:04 a.m., Venkat Ranganathan wrote:
> > I went through the code. It looks good. Thanks for taking the time to write this feature. I am not sure whether calling it MainframeImportTool is a proper name though. We currently support import from Z/OS DB2 also, so not sure whether this can confuse people. Also, this can be a specialization of a generic ftp record import method. That said, I am OK with the current names and formats if there are no further comments from others.
Very good points Venkat. I'm expecting that FTP based Mainframe connector is only a first step and that others might write more performant Mainframe connectors in the future that will be based on top of this work. Hence I would personally prefer to keep it called "Mainframe" to create similar connector based architecture as we do have for JDBC compliant databases.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52127
-----------------------------------------------------------
On Aug. 27, 2014, 7:55 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2014, 7:55 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-connection-args.txt PRE-CREATION
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop-import.txt 00b1ec8
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
> On Sept. 3, 2014, 5:04 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/SqoopOptions.java, line 2176
> > <https://reviews.apache.org/r/22516/diff/2/?file=670466#file670466line2176>
> >
> > Do we need to set tablename here - is it to overcome validations?
Yes, it is to make sure that nothing breaks.
> On Sept. 3, 2014, 5:04 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java, line 131
> > <https://reviews.apache.org/r/22516/diff/2/?file=670477#file670477line131>
> >
> > Why can't we throw the IOExceptin and a finally clause to wrap the entire body of code to also disconnect from the FTP server. That will be clean and will cover all exit cases
I totally agree with you. I made the necessary code changes.
> On Sept. 3, 2014, 5:04 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java, line 145
> > <https://reviews.apache.org/r/22516/diff/2/?file=670477#file670477line145>
> >
> > Same as above.
Fixed.
> On Sept. 3, 2014, 5:04 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java, line 154
> > <https://reviews.apache.org/r/22516/diff/2/?file=670477#file670477line154>
> >
> > Throwing the exception here with the ioe as the cause would provide better diagnostics in stack traces (For example,
> > throw new IOException("....", ioe);
Fixed.
- Mariappan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52127
-----------------------------------------------------------
On Aug. 27, 2014, 7:55 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2014, 7:55 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-connection-args.txt PRE-CREATION
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop-import.txt 00b1ec8
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52127
-----------------------------------------------------------
I went through the code. It looks good. Thanks for taking the time to write this feature. I am not sure whether calling it MainframeImportTool is a proper name though. We currently support import from Z/OS DB2 also, so not sure whether this can confuse people. Also, this can be a specialization of a generic ftp record import method. That said, I am OK with the current names and formats if there are no further comments from others.
src/java/org/apache/sqoop/SqoopOptions.java
<https://reviews.apache.org/r/22516/#comment90872>
Do we need to set tablename here - is it to overcome validations?
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java
<https://reviews.apache.org/r/22516/#comment90873>
Why can't we throw the IOExceptin and a finally clause to wrap the entire body of code to also disconnect from the FTP server. That will be clean and will cover all exit cases
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java
<https://reviews.apache.org/r/22516/#comment90874>
Same as above.
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java
<https://reviews.apache.org/r/22516/#comment90875>
Throwing the exception here with the ioe as the cause would provide better diagnostics in stack traces (For example,
throw new IOException("....", ioe);
- Venkat Ranganathan
On Aug. 27, 2014, 7:55 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Aug. 27, 2014, 7:55 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-connection-args.txt PRE-CREATION
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop-import.txt 00b1ec8
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52706
-----------------------------------------------------------
Ship it!
Hi Asokan,
the changes looks good to me. Please upload latest patch to the JIRA, so that we can commit it. I'll wait couple of days before committing to allow other reviewiers finish their reviews.
Jarcec
- Jarek Cecho
On Sept. 8, 2014, 11:19 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2014, 11:19 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-independent-args.txt PRE-CREATION
> src/docs/man/hbase-args.txt 456bc14
> src/docs/man/hive-args.txt dd77c3e
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Venkat Ranganathan <n....@live.com>.
> On Sept. 9, 2014, 4:51 p.m., Venkat Ranganathan wrote:
> > I build the docs and ran through the tests. It looks good. Thanks for taking all the feedback in. +1.
> >
> > No more review comments presently from me.
> >
> > Venkat
I meant I built the docs.. Sorry for the typo
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52737
-----------------------------------------------------------
On Sept. 8, 2014, 11:19 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2014, 11:19 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-independent-args.txt PRE-CREATION
> src/docs/man/hbase-args.txt 456bc14
> src/docs/man/hive-args.txt dd77c3e
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52737
-----------------------------------------------------------
Ship it!
I build the docs and ran through the tests. It looks good. Thanks for taking all the feedback in. +1.
No more review comments presently from me.
Venkat
- Venkat Ranganathan
On Sept. 8, 2014, 11:19 p.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2014, 11:19 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-independent-args.txt PRE-CREATION
> src/docs/man/hbase-args.txt 456bc14
> src/docs/man/hive-args.txt dd77c3e
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/#review52819
-----------------------------------------------------------
Ship it!
Thanks Mariappan for updating and testing in the real scenario. Looked at the new changes in the 2 files and look good.
- Venkat Ranganathan
On Sept. 10, 2014, 1:33 a.m., Mariappan Asokan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22516/
> -----------------------------------------------------------
>
> (Updated Sept. 10, 2014, 1:33 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> This is to move mainframe datasets to Hadoop.
>
>
> Diffs
> -----
>
> ivy.xml 6335e01
> ivy/libraries.properties 6818b3e
> src/docs/man/common-args.txt e8d1f17
> src/docs/man/database-independent-args.txt PRE-CREATION
> src/docs/man/hbase-args.txt 456bc14
> src/docs/man/hive-args.txt dd77c3e
> src/docs/man/import-args.txt 2bb69ba
> src/docs/man/import-common-args.txt PRE-CREATION
> src/docs/man/mainframe-connection-args.txt PRE-CREATION
> src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
> src/docs/man/sqoop.txt febe827
> src/docs/user/SqoopUserGuide.txt 2e88887
> src/docs/user/basics.txt 7e5a76a
> src/docs/user/connecting-to-mainframe.txt PRE-CREATION
> src/docs/user/distributed-cache.txt PRE-CREATION
> src/docs/user/import-mainframe-purpose.txt PRE-CREATION
> src/docs/user/import-mainframe.txt PRE-CREATION
> src/docs/user/import.txt c5ffa50
> src/docs/user/intro.txt 99cd475
> src/docs/user/mainframe-common-args.txt PRE-CREATION
> src/docs/user/tools.txt 7d977d4
> src/docs/user/validation-args.txt 3cb5f66
> src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
> src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
> src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
> src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
> src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
> src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
> src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
> src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
> src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/22516/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Mariappan Asokan
>
>
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/
-----------------------------------------------------------
(Updated Sept. 10, 2014, 1:33 a.m.)
Review request for Sqoop.
Changes
-------
isORMFacilitySelfManaged() was removed from MainframeManager.java
Repository: sqoop-trunk
Description
-------
This is to move mainframe datasets to Hadoop.
Diffs (updated)
-----
ivy.xml 6335e01
ivy/libraries.properties 6818b3e
src/docs/man/common-args.txt e8d1f17
src/docs/man/database-independent-args.txt PRE-CREATION
src/docs/man/hbase-args.txt 456bc14
src/docs/man/hive-args.txt dd77c3e
src/docs/man/import-args.txt 2bb69ba
src/docs/man/import-common-args.txt PRE-CREATION
src/docs/man/mainframe-connection-args.txt PRE-CREATION
src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
src/docs/man/sqoop.txt febe827
src/docs/user/SqoopUserGuide.txt 2e88887
src/docs/user/basics.txt 7e5a76a
src/docs/user/connecting-to-mainframe.txt PRE-CREATION
src/docs/user/distributed-cache.txt PRE-CREATION
src/docs/user/import-mainframe-purpose.txt PRE-CREATION
src/docs/user/import-mainframe.txt PRE-CREATION
src/docs/user/import.txt c5ffa50
src/docs/user/intro.txt 99cd475
src/docs/user/mainframe-common-args.txt PRE-CREATION
src/docs/user/tools.txt 7d977d4
src/docs/user/validation-args.txt 3cb5f66
src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
Diff: https://reviews.apache.org/r/22516/diff/
Testing
-------
Thanks,
Mariappan Asokan
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/
-----------------------------------------------------------
(Updated Sept. 8, 2014, 11:19 p.m.)
Review request for Sqoop.
Repository: sqoop-trunk
Description
-------
This is to move mainframe datasets to Hadoop.
Diffs (updated)
-----
ivy.xml 6335e01
ivy/libraries.properties 6818b3e
src/docs/man/common-args.txt e8d1f17
src/docs/man/database-independent-args.txt PRE-CREATION
src/docs/man/hbase-args.txt 456bc14
src/docs/man/hive-args.txt dd77c3e
src/docs/man/import-args.txt 2bb69ba
src/docs/man/import-common-args.txt PRE-CREATION
src/docs/man/mainframe-connection-args.txt PRE-CREATION
src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
src/docs/man/sqoop.txt febe827
src/docs/user/SqoopUserGuide.txt 2e88887
src/docs/user/basics.txt 7e5a76a
src/docs/user/connecting-to-mainframe.txt PRE-CREATION
src/docs/user/distributed-cache.txt PRE-CREATION
src/docs/user/import-mainframe-purpose.txt PRE-CREATION
src/docs/user/import-mainframe.txt PRE-CREATION
src/docs/user/import.txt c5ffa50
src/docs/user/intro.txt 99cd475
src/docs/user/mainframe-common-args.txt PRE-CREATION
src/docs/user/tools.txt 7d977d4
src/docs/user/validation-args.txt 3cb5f66
src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
Diff: https://reviews.apache.org/r/22516/diff/
Testing
-------
Thanks,
Mariappan Asokan
Re: Review Request 22516: Support importing mainframe sequential
datasets
Posted by Mariappan Asokan <ma...@syncsort.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22516/
-----------------------------------------------------------
(Updated Aug. 27, 2014, 7:55 p.m.)
Review request for Sqoop.
Repository: sqoop-trunk
Description
-------
This is to move mainframe datasets to Hadoop.
Diffs (updated)
-----
ivy.xml 6335e01
ivy/libraries.properties 6818b3e
src/docs/man/common-args.txt e8d1f17
src/docs/man/database-connection-args.txt PRE-CREATION
src/docs/man/import-args.txt 2bb69ba
src/docs/man/import-common-args.txt PRE-CREATION
src/docs/man/mainframe-connection-args.txt PRE-CREATION
src/docs/man/sqoop-import-mainframe.txt PRE-CREATION
src/docs/man/sqoop-import.txt 00b1ec8
src/docs/man/sqoop.txt febe827
src/docs/user/SqoopUserGuide.txt 2e88887
src/docs/user/basics.txt 7e5a76a
src/docs/user/connecting-to-mainframe.txt PRE-CREATION
src/docs/user/distributed-cache.txt PRE-CREATION
src/docs/user/import-mainframe-purpose.txt PRE-CREATION
src/docs/user/import-mainframe.txt PRE-CREATION
src/docs/user/import.txt c5ffa50
src/docs/user/intro.txt 99cd475
src/docs/user/mainframe-common-args.txt PRE-CREATION
src/docs/user/tools.txt 7d977d4
src/docs/user/validation-args.txt 3cb5f66
src/java/org/apache/sqoop/SqoopOptions.java 3ef5a97
src/java/org/apache/sqoop/manager/MainframeManager.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputFormat.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetInputSplit.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetRecordReader.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java PRE-CREATION
src/java/org/apache/sqoop/tool/MainframeImportTool.java PRE-CREATION
src/java/org/apache/sqoop/tool/SqoopTool.java dbe429a
src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java PRE-CREATION
src/test/org/apache/sqoop/manager/TestMainframeManager.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputSplit.java PRE-CREATION
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeImportJob.java PRE-CREATION
src/test/org/apache/sqoop/tool/TestMainframeImportTool.java PRE-CREATION
src/test/org/apache/sqoop/util/TestMainframeFTPClientUtils.java PRE-CREATION
Diff: https://reviews.apache.org/r/22516/diff/
Testing
-------
Thanks,
Mariappan Asokan