You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jimmy Xiang <jx...@cloudera.com> on 2014/11/04 19:58:53 UTC
Review Request 27580: HIVE-8623 Implement HashTableLoader for Spark
map-join [Spark Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27580/
-----------------------------------------------------------
Review request for hive and Xuefu Zhang.
Bugs: HIVE-8623
https://issues.apache.org/jira/browse/HIVE-8623
Repository: hive-git
Description
-------
Loading HashTable for Spark map-join. It's assumed that all tables share the same base dir. Each table has its own sub-folder. There could be several HashTable files for each table.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 10ad933
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java da36848
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java PRE-CREATION
Diff: https://reviews.apache.org/r/27580/diff/
Testing
-------
Thanks,
Jimmy Xiang
Re: Review Request 27580: HIVE-8623 Implement HashTableLoader for
Spark map-join [Spark Branch]
Posted by Jimmy Xiang <jx...@cloudera.com>.
> On Nov. 4, 2014, 8:38 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java, line 123
> > <https://reviews.apache.org/r/27580/diff/1/?file=748415#file748415line123>
> >
> > Nit: There is potential fd leak in case fs.open is successful while new ObjectInputStream() fails.
Good catch. Fixed in v3.
- Jimmy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27580/#review59823
-----------------------------------------------------------
On Nov. 4, 2014, 9:02 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27580/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2014, 9:02 p.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-8623
> https://issues.apache.org/jira/browse/HIVE-8623
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Loading HashTable for Spark map-join. It's assumed that all tables share the same base dir. Each table has its own sub-folder. There could be several HashTable files for each table.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 10ad933
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java da36848
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27580/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 27580: HIVE-8623 Implement HashTableLoader for
Spark map-join [Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27580/#review59823
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
<https://reviews.apache.org/r/27580/#comment101118>
Nit: There is potential fd leak in case fs.open is successful while new ObjectInputStream() fails.
- Xuefu Zhang
On Nov. 4, 2014, 6:58 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27580/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2014, 6:58 p.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-8623
> https://issues.apache.org/jira/browse/HIVE-8623
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Loading HashTable for Spark map-join. It's assumed that all tables share the same base dir. Each table has its own sub-folder. There could be several HashTable files for each table.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 10ad933
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java da36848
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27580/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 27580: HIVE-8623 Implement HashTableLoader for
Spark map-join [Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27580/#review59871
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Nov. 4, 2014, 9:02 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27580/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2014, 9:02 p.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-8623
> https://issues.apache.org/jira/browse/HIVE-8623
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Loading HashTable for Spark map-join. It's assumed that all tables share the same base dir. Each table has its own sub-folder. There could be several HashTable files for each table.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 10ad933
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java da36848
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/27580/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 27580: HIVE-8623 Implement HashTableLoader for
Spark map-join [Spark Branch]
Posted by Jimmy Xiang <jx...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27580/
-----------------------------------------------------------
(Updated Nov. 4, 2014, 9:02 p.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
Addressed the fd leak issue Xuefu pointed out.
Bugs: HIVE-8623
https://issues.apache.org/jira/browse/HIVE-8623
Repository: hive-git
Description
-------
Loading HashTable for Spark map-join. It's assumed that all tables share the same base dir. Each table has its own sub-folder. There could be several HashTable files for each table.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java 10ad933
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java da36848
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java PRE-CREATION
Diff: https://reviews.apache.org/r/27580/diff/
Testing
-------
Thanks,
Jimmy Xiang