You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2016/03/02 03:30:39 UTC
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
-----------------------------------------------------------
(Updated March 2, 2016, 2:30 a.m.)
Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java b5c4d1d
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java cb092d1
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
Diff: https://reviews.apache.org/r/40415/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On March 7, 2016, 7:27 a.m., Prasanth_J wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java, line 2223
> > <https://reviews.apache.org/r/40415/diff/6/?file=1276199#file1276199line2223>
> >
> > use ThreadLocal?
Not sure that helps. Also, it's a threadlocal... Remembering the result for some time could be a good solution in a follow-up, but we assume by default that if the user enables it they know what they are doing, or pay a small perf penalty
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/#review122261
-----------------------------------------------------------
On March 2, 2016, 2:30 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40415/
> -----------------------------------------------------------
>
> (Updated March 2, 2016, 2:30 a.m.)
>
>
> Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
> metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
> metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java b5c4d1d
> metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java cb092d1
> metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
> orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
>
> Diff: https://reviews.apache.org/r/40415/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/#review122261
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java (line 2222)
<https://reviews.apache.org/r/40415/#comment184177>
use ThreadLocal?
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java (line 2251)
<https://reviews.apache.org/r/40415/#comment184178>
Can this be set here or provide a separate API for not getting footer? We would otherwise transfer more data through wire.
orc/src/java/org/apache/orc/impl/InStream.java
<https://reviews.apache.org/r/40415/#comment184179>
nit: Unrelated changes.
- Prasanth_J
On March 2, 2016, 2:30 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40415/
> -----------------------------------------------------------
>
> (Updated March 2, 2016, 2:30 a.m.)
>
>
> Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
> metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
> metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java b5c4d1d
> metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java cb092d1
> metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
> orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
>
> Diff: https://reviews.apache.org/r/40415/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
-----------------------------------------------------------
(Updated March 9, 2016, 2:07 a.m.)
Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851
metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 0e8a157
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 9048d45
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 62677d1
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java fdc7956
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
Diff: https://reviews.apache.org/r/40415/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/#review122384
-----------------------------------------------------------
Ship it!
Ship It!
- Prasanth_J
On March 7, 2016, 9:36 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40415/
> -----------------------------------------------------------
>
> (Updated March 7, 2016, 9:36 p.m.)
>
>
> Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851
> metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
> metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java b5c4d1d
> metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java cb092d1
> metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
> orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
> ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ad17096
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
> storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
>
> Diff: https://reviews.apache.org/r/40415/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
-----------------------------------------------------------
(Updated March 7, 2016, 9:36 p.m.)
Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851
metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java b5c4d1d
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java cb092d1
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ad17096
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
Diff: https://reviews.apache.org/r/40415/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40415: HIVE-11675 make use of file footer PPD API
in ETL strategy or separate strategy
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40415/
-----------------------------------------------------------
(Updated March 7, 2016, 9:32 p.m.)
Review request for hive, Gopal V, Prasanth_J, and Vikram Dixit Kumaraswamy.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851
metastore/src/java/org/apache/hadoop/hive/metastore/FileFormatProxy.java ec0be2b
metastore/src/java/org/apache/hadoop/hive/metastore/FileMetadataHandler.java bd4e188
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java b5c4d1d
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java cb092d1
metastore/src/java/org/apache/hadoop/hive/metastore/filemeta/OrcFileMetadataHandler.java 1b388aa
orc/src/java/org/apache/orc/impl/InStream.java b1c6de5
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 9bf9377
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/orc/ExternalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetastoreExternalCachesByConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileFormatProxy.java ef76723
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewInputFormat.java 2782d7e
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ad17096
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSplitElimination.java 7a93b54
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/PredicateLeaf.java dc71db4
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java be5e67b
Diff: https://reviews.apache.org/r/40415/diff/
Testing
-------
Thanks,
Sergey Shelukhin