You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Abraham Fine <ab...@abrahamfine.com> on 2016/02/26 05:48:06 UTC
Review Request 44054: Sqoop2: Job failure when writing parquet in
hdfs with data coming from mysql
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44054/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-2849
https://issues.apache.org/jira/browse/SQOOP-2849
Repository: sqoop-sqoop2
Description
-------
Sqoop2: Job failure when writing parquet in hdfs with data coming from mysql
Diffs
-----
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java 89bc0f2cce7d07e4e8076a623d71f40edabedd40
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java b55f7a00f5ae6156f75c7a166b1fd5303b3976a6
connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestAVROIntermediateDataFormat.java 3c4d7ded81b097f1e242b4ed7eed7d0b546158d8
Diff: https://reviews.apache.org/r/44054/diff/
Testing
-------
yes
Thanks,
Abraham Fine
Re: Review Request 44054: Sqoop2: Job failure when writing parquet in
hdfs with data coming from mysql
Posted by Abraham Fine <ab...@abrahamfine.com>.
> On Feb. 26, 2016, 4:52 a.m., Jarek Cecho wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java, lines 69-73
> > <https://reviews.apache.org/r/44054/diff/1/?file=1271818#file1271818line69>
> >
> > My sense is that this method can create duplicates, e.g. column names "jarcec" and "jar cec" will lead to the same. Should we add a verification code that will append 1, 2, 3, ... in case that we will create duplicates?
The issue of duplicates is interesting. I did not want to append anything but I thought logging something would be sufficient for now. My understanding was that in the future we would have column name mapping to deal with this.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44054/#review120824
-----------------------------------------------------------
On Feb. 26, 2016, 4:48 a.m., Abraham Fine wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44054/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2016, 4:48 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2849
> https://issues.apache.org/jira/browse/SQOOP-2849
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Sqoop2: Job failure when writing parquet in hdfs with data coming from mysql
>
>
> Diffs
> -----
>
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java 89bc0f2cce7d07e4e8076a623d71f40edabedd40
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java b55f7a00f5ae6156f75c7a166b1fd5303b3976a6
> connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestAVROIntermediateDataFormat.java 3c4d7ded81b097f1e242b4ed7eed7d0b546158d8
>
> Diff: https://reviews.apache.org/r/44054/diff/
>
>
> Testing
> -------
>
> yes
>
>
> Thanks,
>
> Abraham Fine
>
>
Re: Review Request 44054: Sqoop2: Job failure when writing parquet in
hdfs with data coming from mysql
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44054/#review120824
-----------------------------------------------------------
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java (lines 69 - 73)
<https://reviews.apache.org/r/44054/#comment182331>
My sense is that this method can create duplicates, e.g. column names "jarcec" and "jar cec" will lead to the same. Should we add a verification code that will append 1, 2, 3, ... in case that we will create duplicates?
- Jarek Cecho
On Feb. 26, 2016, 4:48 a.m., Abraham Fine wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44054/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2016, 4:48 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2849
> https://issues.apache.org/jira/browse/SQOOP-2849
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Sqoop2: Job failure when writing parquet in hdfs with data coming from mysql
>
>
> Diffs
> -----
>
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java 89bc0f2cce7d07e4e8076a623d71f40edabedd40
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java b55f7a00f5ae6156f75c7a166b1fd5303b3976a6
> connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestAVROIntermediateDataFormat.java 3c4d7ded81b097f1e242b4ed7eed7d0b546158d8
>
> Diff: https://reviews.apache.org/r/44054/diff/
>
>
> Testing
> -------
>
> yes
>
>
> Thanks,
>
> Abraham Fine
>
>
Re: Review Request 44054: Sqoop2: Job failure when writing parquet in
hdfs with data coming from mysql
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44054/#review121307
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Feb. 26, 2016, 6:48 p.m., Abraham Fine wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44054/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2016, 6:48 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-2849
> https://issues.apache.org/jira/browse/SQOOP-2849
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Sqoop2: Job failure when writing parquet in hdfs with data coming from mysql
>
>
> Diffs
> -----
>
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java 89bc0f2cce7d07e4e8076a623d71f40edabedd40
> connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java b55f7a00f5ae6156f75c7a166b1fd5303b3976a6
> connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestAVROIntermediateDataFormat.java 3c4d7ded81b097f1e242b4ed7eed7d0b546158d8
>
> Diff: https://reviews.apache.org/r/44054/diff/
>
>
> Testing
> -------
>
> yes
>
>
> Thanks,
>
> Abraham Fine
>
>
Re: Review Request 44054: Sqoop2: Job failure when writing parquet in
hdfs with data coming from mysql
Posted by Abraham Fine <ab...@abrahamfine.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44054/
-----------------------------------------------------------
(Updated Feb. 26, 2016, 6:48 p.m.)
Review request for Sqoop.
Bugs: SQOOP-2849
https://issues.apache.org/jira/browse/SQOOP-2849
Repository: sqoop-sqoop2
Description
-------
Sqoop2: Job failure when writing parquet in hdfs with data coming from mysql
Diffs (updated)
-----
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/common/SqoopAvroUtils.java 89bc0f2cce7d07e4e8076a623d71f40edabedd40
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/AVROIntermediateDataFormat.java b55f7a00f5ae6156f75c7a166b1fd5303b3976a6
connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestAVROIntermediateDataFormat.java 3c4d7ded81b097f1e242b4ed7eed7d0b546158d8
Diff: https://reviews.apache.org/r/44054/diff/
Testing
-------
yes
Thanks,
Abraham Fine