You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Colin Ma <ju...@intel.com> on 2015/08/21 09:54:53 UTC
Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/
-----------------------------------------------------------
Review request for Sqoop.
Repository: sqoop-sqoop2
Description
-------
Fix warning in execution module
Diffs
-----
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
Diff: https://reviews.apache.org/r/37672/diff/
Testing
-------
Thanks,
Colin Ma
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Jarek Cecho <ja...@apache.org>.
> On Aug. 21, 2015, 9 p.m., Abraham Elmahrek wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java, line 94
> > <https://reviews.apache.org/r/37672/diff/1/?file=1046557#file1046557line94>
> >
> > Charset.defaultCharset()
Hi Abe,
thank you for the comment. We've started using UTF-8 for string serialization in SQOOP-2451. I've picked UTF-8 there as that was something we were assuming under the hood.
My reasoning for not using defaultCharset() is that it can be different on different nodes (different slave nodes on the Hadoop cluster, Server and the client). I wanted to explicitly want to avoid situation when one node (let say Sqoop server) will serialize data with UTF-8, so that other node (let say mapper) will de-serialize them with different default encoding.
Do you think that it make sense or would you prefer us to go back and use Charset.defaultCharset() elsewhere as well?
Jarcec
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/#review96082
-----------------------------------------------------------
On Aug. 21, 2015, 7:54 a.m., Colin Ma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37672/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2015, 7:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Fix warning in execution module
>
>
> Diffs
> -----
>
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
>
> Diff: https://reviews.apache.org/r/37672/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Colin Ma
>
>
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Jarek Cecho <ja...@apache.org>.
> On Aug. 21, 2015, 9 p.m., Abraham Elmahrek wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java, line 94
> > <https://reviews.apache.org/r/37672/diff/1/?file=1046557#file1046557line94>
> >
> > Charset.defaultCharset()
>
> Jarek Cecho wrote:
> Hi Abe,
> thank you for the comment. We've started using UTF-8 for string serialization in SQOOP-2451. I've picked UTF-8 there as that was something we were assuming under the hood.
>
> My reasoning for not using defaultCharset() is that it can be different on different nodes (different slave nodes on the Hadoop cluster, Server and the client). I wanted to explicitly want to avoid situation when one node (let say Sqoop server) will serialize data with UTF-8, so that other node (let say mapper) will de-serialize them with different default encoding.
>
> Do you think that it make sense or would you prefer us to go back and use Charset.defaultCharset() elsewhere as well?
>
> Jarcec
>
> Abraham Elmahrek wrote:
> UTF-8 makes sense to me. It is different functionality than what was initially provided. Calling getBytes without a charset will use the default encoding. I'm cool with using UTF-8 every where though.
Yeah you're right - we're changing the functionality here. We should have been more explicit that we're doing this conversion on purpose.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/#review96082
-----------------------------------------------------------
On Aug. 21, 2015, 7:54 a.m., Colin Ma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37672/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2015, 7:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Fix warning in execution module
>
>
> Diffs
> -----
>
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
>
> Diff: https://reviews.apache.org/r/37672/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Colin Ma
>
>
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Colin Ma <ju...@intel.com>.
> On Aug. 21, 2015, 9 p.m., Abraham Elmahrek wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java, line 58
> > <https://reviews.apache.org/r/37672/diff/1/?file=1046559#file1046559line58>
> >
> > Private as well.
This will be updated.
> On Aug. 21, 2015, 9 p.m., Abraham Elmahrek wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java, line 76
> > <https://reviews.apache.org/r/37672/diff/1/?file=1046560#file1046560line76>
> >
> > Might not be necessary if we create a constructor that consumes a partition?
> >
> > Also, error handling is missing.
I'll create a constructor, thanks.
- Colin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/#review96082
-----------------------------------------------------------
On Aug. 21, 2015, 7:54 a.m., Colin Ma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37672/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2015, 7:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Fix warning in execution module
>
>
> Diffs
> -----
>
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
>
> Diff: https://reviews.apache.org/r/37672/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Colin Ma
>
>
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On Aug. 21, 2015, 9 p.m., Abraham Elmahrek wrote:
> > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java, line 94
> > <https://reviews.apache.org/r/37672/diff/1/?file=1046557#file1046557line94>
> >
> > Charset.defaultCharset()
>
> Jarek Cecho wrote:
> Hi Abe,
> thank you for the comment. We've started using UTF-8 for string serialization in SQOOP-2451. I've picked UTF-8 there as that was something we were assuming under the hood.
>
> My reasoning for not using defaultCharset() is that it can be different on different nodes (different slave nodes on the Hadoop cluster, Server and the client). I wanted to explicitly want to avoid situation when one node (let say Sqoop server) will serialize data with UTF-8, so that other node (let say mapper) will de-serialize them with different default encoding.
>
> Do you think that it make sense or would you prefer us to go back and use Charset.defaultCharset() elsewhere as well?
>
> Jarcec
UTF-8 makes sense to me. It is different functionality than what was initially provided. Calling getBytes without a charset will use the default encoding. I'm cool with using UTF-8 every where though.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/#review96082
-----------------------------------------------------------
On Aug. 21, 2015, 7:54 a.m., Colin Ma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37672/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2015, 7:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Fix warning in execution module
>
>
> Diffs
> -----
>
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
>
> Diff: https://reviews.apache.org/r/37672/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Colin Ma
>
>
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/#review96082
-----------------------------------------------------------
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 94)
<https://reviews.apache.org/r/37672/#comment151289>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 99)
<https://reviews.apache.org/r/37672/#comment151290>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 114)
<https://reviews.apache.org/r/37672/#comment151291>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 119)
<https://reviews.apache.org/r/37672/#comment151293>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 133)
<https://reviews.apache.org/r/37672/#comment151292>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 147)
<https://reviews.apache.org/r/37672/#comment151294>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 150)
<https://reviews.apache.org/r/37672/#comment151295>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 237)
<https://reviews.apache.org/r/37672/#comment151296>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java (line 257)
<https://reviews.apache.org/r/37672/#comment151297>
Charset.defaultCharset()
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java (line 58)
<https://reviews.apache.org/r/37672/#comment151298>
Private as well.
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java (line 76)
<https://reviews.apache.org/r/37672/#comment151299>
Might not be necessary if we create a constructor that consumes a partition?
Also, error handling is missing.
- Abraham Elmahrek
On Aug. 21, 2015, 7:54 a.m., Colin Ma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37672/
> -----------------------------------------------------------
>
> (Updated Aug. 21, 2015, 7:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Fix warning in execution module
>
>
> Diffs
> -----
>
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
>
> Diff: https://reviews.apache.org/r/37672/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Colin Ma
>
>
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/#review96349
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Aug. 25, 2015, 1:36 a.m., Colin Ma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37672/
> -----------------------------------------------------------
>
> (Updated Aug. 25, 2015, 1:36 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> Fix warning in execution module
>
>
> Diffs
> -----
>
> execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 5aef878
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
> execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
>
> Diff: https://reviews.apache.org/r/37672/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Colin Ma
>
>
Re: Review Request 37672: SQOOP-2516: Sqoop2: Findbugs: Fix warning in
execution module
Posted by Colin Ma <ju...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37672/
-----------------------------------------------------------
(Updated Aug. 25, 2015, 1:36 a.m.)
Review request for Sqoop.
Repository: sqoop-sqoop2
Description
-------
Fix warning in execution module
Diffs (updated)
-----
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 3f79325
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/MRConfigurationUtils.java e07ddd7
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 5aef878
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 14fdfdc
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 6134106
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java b81f4d6
Diff: https://reviews.apache.org/r/37672/diff/
Testing
-------
Thanks,
Colin Ma