You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Boglarka Egyed <bo...@apache.org> on 2017/11/06 12:27:32 UTC
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/
-----------------------------------------------------------
(Updated Nov. 6, 2017, 12:27 p.m.)
Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
Summary (updated)
-----------------
SQOOP-3243: Importing BLOB data causes "Stream closed" error on encrypted HDFS
Bugs: SQOOP-3243
https://issues.apache.org/jira/browse/SQOOP-3243
Repository: sqoop-trunk
Description
-------
Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream butis propagated to upper levels in Java 8 thus the error occurs with BLOB import.
Diffs
-----
ivy.xml 601aa015c5052da1a287baec07cea4c26e143594
ivy/libraries.properties 2ef04f4fdeca4b6d82d306870cc715fbb98e56d4
src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java 73b31772d73594536f93b71ad9639c4d1a3e34e5
src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java 250ffa640b63fe06635a93895e2fe3c8fedfbbaa
src/test/org/apache/sqoop/tool/TestImportTool.java 6335fb1aa3ff63c12ae82cb69e3f77ade3c48c53
Diff: https://reviews.apache.org/r/63576/diff/1/
Testing (updated)
-------
I have added a new test case, I also needed to bump the Mockito version and upon that some other test cases needed to be updated as well due to some Mockito backward incompatibility issues.
Ran unit and 3rd party tests successfully.
Thanks,
Boglarka Egyed
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Boglarka Egyed <bo...@apache.org>.
> On Nov. 6, 2017, 2:03 p.m., Szabolcs Vasas wrote:
> > src/test/com/cloudera/sqoop/io/TestLobFile.java
> > Lines 615 (patched)
> > <https://reviews.apache.org/r/63576/diff/2/?file=1881914#file1881914line615>
> >
> > I think if you use the spy method here just like you did in the createCryptoOutputStream method you can avoid the library upgrade.
This is indeed a more elegant solution, thanks for the suggestion!
- Boglarka
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/#review190153
-----------------------------------------------------------
On Nov. 6, 2017, 2:46 p.m., Boglarka Egyed wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63576/
> -----------------------------------------------------------
>
> (Updated Nov. 6, 2017, 2:46 p.m.)
>
>
> Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
>
>
> Bugs: SQOOP-3243
> https://issues.apache.org/jira/browse/SQOOP-3243
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream butis propagated to upper levels in Java 8 thus the error occurs with BLOB import.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
> src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
>
>
> Diff: https://reviews.apache.org/r/63576/diff/3/
>
>
> Testing
> -------
>
> I have added a new test case, I also needed to bump the Mockito version and upon that some other test cases needed to be updated as well due to some Mockito backward incompatibility issues.
> Ran unit and 3rd party tests successfully.
>
>
> Thanks,
>
> Boglarka Egyed
>
>
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/#review190153
-----------------------------------------------------------
Hi Bogi,
Nice catch!
I left a comment, I think the library upgrade can be avoided so we can make this patch simpler, please take a look.
Szabolcs
src/test/com/cloudera/sqoop/io/TestLobFile.java
Lines 615 (patched)
<https://reviews.apache.org/r/63576/#comment267397>
I think if you use the spy method here just like you did in the createCryptoOutputStream method you can avoid the library upgrade.
- Szabolcs Vasas
On Nov. 6, 2017, 12:58 p.m., Boglarka Egyed wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63576/
> -----------------------------------------------------------
>
> (Updated Nov. 6, 2017, 12:58 p.m.)
>
>
> Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
>
>
> Bugs: SQOOP-3243
> https://issues.apache.org/jira/browse/SQOOP-3243
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream butis propagated to upper levels in Java 8 thus the error occurs with BLOB import.
>
>
> Diffs
> -----
>
> ivy.xml 601aa015c5052da1a287baec07cea4c26e143594
> ivy/libraries.properties 2ef04f4fdeca4b6d82d306870cc715fbb98e56d4
> src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
> src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
> src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java 73b31772d73594536f93b71ad9639c4d1a3e34e5
> src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java 250ffa640b63fe06635a93895e2fe3c8fedfbbaa
> src/test/org/apache/sqoop/tool/TestImportTool.java 6335fb1aa3ff63c12ae82cb69e3f77ade3c48c53
>
>
> Diff: https://reviews.apache.org/r/63576/diff/2/
>
>
> Testing
> -------
>
> I have added a new test case, I also needed to bump the Mockito version and upon that some other test cases needed to be updated as well due to some Mockito backward incompatibility issues.
> Ran unit and 3rd party tests successfully.
>
>
> Thanks,
>
> Boglarka Egyed
>
>
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Szabolcs Vasas <va...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/#review190179
-----------------------------------------------------------
Ship it!
Thank you for improving the patch, it looks good to me now.
- Szabolcs Vasas
On Nov. 6, 2017, 4:28 p.m., Boglarka Egyed wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63576/
> -----------------------------------------------------------
>
> (Updated Nov. 6, 2017, 4:28 p.m.)
>
>
> Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
>
>
> Bugs: SQOOP-3243
> https://issues.apache.org/jira/browse/SQOOP-3243
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream but is propagated to upper levels in Java 8 thus the error occurs with BLOB import.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
> src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
>
>
> Diff: https://reviews.apache.org/r/63576/diff/4/
>
>
> Testing
> -------
>
> I have added a new test case and run unit and 3rd party tests successfully.
>
>
> Thanks,
>
> Boglarka Egyed
>
>
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/
-----------------------------------------------------------
(Updated Nov. 6, 2017, 4:28 p.m.)
Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
Changes
-------
Exchange deprecated FSDataOutputStream constructor usage.
Bugs: SQOOP-3243
https://issues.apache.org/jira/browse/SQOOP-3243
Repository: sqoop-trunk
Description
-------
Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream but is propagated to upper levels in Java 8 thus the error occurs with BLOB import.
Diffs (updated)
-----
src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
Diff: https://reviews.apache.org/r/63576/diff/4/
Changes: https://reviews.apache.org/r/63576/diff/3-4/
Testing
-------
I have added a new test case and run unit and 3rd party tests successfully.
Thanks,
Boglarka Egyed
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/
-----------------------------------------------------------
(Updated Nov. 6, 2017, 3:16 p.m.)
Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
Bugs: SQOOP-3243
https://issues.apache.org/jira/browse/SQOOP-3243
Repository: sqoop-trunk
Description
-------
Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream but is propagated to upper levels in Java 8 thus the error occurs with BLOB import.
Diffs
-----
src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
Diff: https://reviews.apache.org/r/63576/diff/3/
Testing (updated)
-------
I have added a new test case and run unit and 3rd party tests successfully.
Thanks,
Boglarka Egyed
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/
-----------------------------------------------------------
(Updated Nov. 6, 2017, 3:12 p.m.)
Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
Bugs: SQOOP-3243
https://issues.apache.org/jira/browse/SQOOP-3243
Repository: sqoop-trunk
Description (updated)
-------
Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream but is propagated to upper levels in Java 8 thus the error occurs with BLOB import.
Diffs
-----
src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
Diff: https://reviews.apache.org/r/63576/diff/3/
Testing
-------
I have added a new test case, I also needed to bump the Mockito version and upon that some other test cases needed to be updated as well due to some Mockito backward incompatibility issues.
Ran unit and 3rd party tests successfully.
Thanks,
Boglarka Egyed
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/
-----------------------------------------------------------
(Updated Nov. 6, 2017, 2:46 p.m.)
Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
Changes
-------
Reverted Mockito version bump.
Bugs: SQOOP-3243
https://issues.apache.org/jira/browse/SQOOP-3243
Repository: sqoop-trunk
Description
-------
Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream butis propagated to upper levels in Java 8 thus the error occurs with BLOB import.
Diffs (updated)
-----
src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
Diff: https://reviews.apache.org/r/63576/diff/3/
Changes: https://reviews.apache.org/r/63576/diff/2-3/
Testing
-------
I have added a new test case, I also needed to bump the Mockito version and upon that some other test cases needed to be updated as well due to some Mockito backward incompatibility issues.
Ran unit and 3rd party tests successfully.
Thanks,
Boglarka Egyed
Re: Review Request 63576: SQOOP-3243: Importing BLOB data causes
"Stream closed" error on encrypted HDFS
Posted by Boglarka Egyed <bo...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63576/
-----------------------------------------------------------
(Updated Nov. 6, 2017, 12:58 p.m.)
Review request for Sqoop, Ferenc Szabo, Anna Szonyi, and Szabolcs Vasas.
Changes
-------
Revert unnecessary method extraction.
Bugs: SQOOP-3243
https://issues.apache.org/jira/browse/SQOOP-3243
Repository: sqoop-trunk
Description
-------
Importing BLOB data causes "Stream closed" error on encrypted HDFS because Sqoop tries to close CryptoOutputStream twice in LobFile class. This error was ignored with Java 7 implementation of FilterOutputStream butis propagated to upper levels in Java 8 thus the error occurs with BLOB import.
Diffs (updated)
-----
ivy.xml 601aa015c5052da1a287baec07cea4c26e143594
ivy/libraries.properties 2ef04f4fdeca4b6d82d306870cc715fbb98e56d4
src/java/org/apache/sqoop/io/LobFile.java 4e1bf2cb4a356ba1a2fc3aeca12859bbeb329cd4
src/test/com/cloudera/sqoop/io/TestLobFile.java 029758c531b0214b9b7f82de59d8fbae61bfdb0a
src/test/org/apache/sqoop/hbase/TestHBasePutProcessor.java 73b31772d73594536f93b71ad9639c4d1a3e34e5
src/test/org/apache/sqoop/mapreduce/TestJdbcExportJob.java 250ffa640b63fe06635a93895e2fe3c8fedfbbaa
src/test/org/apache/sqoop/tool/TestImportTool.java 6335fb1aa3ff63c12ae82cb69e3f77ade3c48c53
Diff: https://reviews.apache.org/r/63576/diff/2/
Changes: https://reviews.apache.org/r/63576/diff/1-2/
Testing
-------
I have added a new test case, I also needed to bump the Mockito version and upon that some other test cases needed to be updated as well due to some Mockito backward incompatibility issues.
Ran unit and 3rd party tests successfully.
Thanks,
Boglarka Egyed