You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Deepak Jaiswal <dj...@hortonworks.com> on 2019/02/05 22:10:01 UTC

Review Request 69903: HIVE-21214

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69903/
-----------------------------------------------------------

Review request for hive and Jason Dere.


Bugs: HIVE-21214
    https://issues.apache.org/jira/browse/HIVE-21214


Repository: hive-git


Description
-------

MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 8937b43811 


Diff: https://reviews.apache.org/r/69903/diff/1/


Testing
-------


Thanks,

Deepak Jaiswal

Re: Review Request 69903: HIVE-21214

Posted by Deepak Jaiswal <dj...@hortonworks.com>.


> On Feb. 5, 2019, 11:53 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Line 1829 (original), 1838 (patched)
> > <https://reviews.apache.org/r/69903/diff/1/?file=2123940#file2123940line1838>
> >
> >     No "if" - this dedup strategy does not work with speculative execution enabled.

Based on my understanding these are the two scenarios,

1. speculative execution succeeds, it has attempt ID 1. The original attempt ID is 0. The logic picks speculative one, regardless of original one's outcome. This works fine.
2. speculative execution fails, throws exception.

Let me know I am getting it wrong.


- Deepak


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69903/#review212581
-----------------------------------------------------------


On Feb. 5, 2019, 10:10 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69903/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2019, 10:10 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Bugs: HIVE-21214
>     https://issues.apache.org/jira/browse/HIVE-21214
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 8937b43811 
> 
> 
> Diff: https://reviews.apache.org/r/69903/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 69903: HIVE-21214

Posted by Jason Dere <jd...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69903/#review212581
-----------------------------------------------------------




ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Line 1829 (original), 1838 (patched)
<https://reviews.apache.org/r/69903/#comment298407>

    No "if" - this dedup strategy does not work with speculative execution enabled.


- Jason Dere


On Feb. 5, 2019, 10:10 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69903/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2019, 10:10 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Bugs: HIVE-21214
>     https://issues.apache.org/jira/browse/HIVE-21214
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 8937b43811 
> 
> 
> Diff: https://reviews.apache.org/r/69903/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 69903: HIVE-21214

Posted by Deepak Jaiswal <dj...@hortonworks.com>.


> On Feb. 5, 2019, 11:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
> > Lines 1876 (patched)
> > <https://reviews.apache.org/r/69903/diff/1/?file=2123940#file2123940line1876>
> >
> >     nit: add the filenames to the error message

will do.


- Deepak


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69903/#review212580
-----------------------------------------------------------


On Feb. 5, 2019, 10:10 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69903/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2019, 10:10 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Bugs: HIVE-21214
>     https://issues.apache.org/jira/browse/HIVE-21214
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 8937b43811 
> 
> 
> Diff: https://reviews.apache.org/r/69903/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 69903: HIVE-21214

Posted by Jason Dere <jd...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69903/#review212580
-----------------------------------------------------------




ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Lines 1876 (patched)
<https://reviews.apache.org/r/69903/#comment298406>

    nit: add the filenames to the error message


- Jason Dere


On Feb. 5, 2019, 10:10 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69903/
> -----------------------------------------------------------
> 
> (Updated Feb. 5, 2019, 10:10 p.m.)
> 
> 
> Review request for hive and Jason Dere.
> 
> 
> Bugs: HIVE-21214
>     https://issues.apache.org/jira/browse/HIVE-21214
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> MoveTask : Use attemptId instead of file size for deduplication of files compareTempOrDuplicateFiles()
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 8937b43811 
> 
> 
> Diff: https://reviews.apache.org/r/69903/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>