Re: Review Request 61882: SQOOP-3215 : sqoop import to hive table as formats(avro, parquet)

[Sandish Kumar HN <sa...@gmail.com>]

> On Sept. 26, 2017, 1:35 p.m., Szabolcs Vasas wrote:
> > Hi Sandish,
> > 
> > Thank you for your patch! Please find my findings below:
> > 
> > - Sqoop uses Kite for importing in Parquet format so I think your patch should also utilize the Kite libraries for Hive table generation (see org.apache.sqoop.mapreduce.ParquetJob#createDataset). This would ensure that the table creation behaviour is the same when we import the data and when we only create the table.
> > - SQLServerHiveImportTest.testNormalHiveImportAsAvro test case fails (SQLServerHiveImportTest is a subclass of TestHiveImport so it has inherited your new test case). Can you please take a look?
> > 
> > I have not had time yet to deeply review the Avro part but my impression is that it needs much more thorough testing(e.g. more data types) since it is a new feature. What do you think?
> > 
> > Regards,
> > Szabolcs

- Do you want me to use org.apache.sqoop.mapreduce.ParquetJob#createDataset in testcases??
- Yes will add more data types on testcases for avro/parquet


> On Sept. 26, 2017, 1:35 p.m., Szabolcs Vasas wrote:
> > src/test/com/cloudera/sqoop/hive/TestHiveImport.java
> > Line 302 (original), 308 (patched)
> > <https://reviews.apache.org/r/61882/diff/1/?file=1802862#file1802862line308>
> >
> >     Why did you need to add this change? As far as I understand your Parquet changes should only be effective if the --create-hive-table option is specified.

Yes I can remove this.


- Sandish Kumar


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61882/#review186246
-----------------------------------------------------------


On Aug. 24, 2017, 9:51 a.m., Sandish Kumar HN wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61882/
> -----------------------------------------------------------
> 
> (Updated Aug. 24, 2017, 9:51 a.m.)
> 
> 
> Review request for Sqoop and Anna Szonyi.
> 
> 
> Bugs: SQOOP-3215
>     https://issues.apache.org/jira/browse/SQOOP-3215
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> sqoop import and create hive table as formats(avro,parquet)
> 
> 
> Diffs
> -----
> 
>   src/java/org/apache/sqoop/hive/TableDefWriter.java deec32d6 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java 1564bdcb 
>   src/test/com/cloudera/sqoop/hive/TestHiveImport.java a624f52b 
>   testdata/hive/scripts/createhiveImportasparquet.q PRE-CREATION 
>   testdata/hive/scripts/normalHiveImportAvro.q PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61882/diff/1/
> 
> 
> Testing
> -------
> 
> testNormalHiveImportAsAvro, testNormalHiveImportAsParquet with table creation checks.
> 
> 
> Thanks,
> 
> Sandish Kumar HN
> 
>