You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Navis Ryu <na...@nexr.com> on 2013/12/04 06:26:32 UTC
Review Request 15993: analyze command failing to collect stats with counter
mechanism
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 142af10
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 0268f98
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 2cc2519
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 960
> > <https://reviews.apache.org/r/15993/diff/6/?file=395986#file395986line960>
> >
> > I don't think we need to call init() from task nodes. Semantics is : init() is called once from client before job is launched. Is there some reason to call init() here now?
It's mistakenly included while testing. But for "jdbc:derby" type with non-local mode execution, it throws exception that "there is no such table", etc. Maybe we might need some handling for this case (ignore?)
> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 970
> > <https://reviews.apache.org/r/15993/diff/6/?file=395986#file395986line970>
> >
> > This can simply be written as
> > String spSpec = conf.getStaticSpec();
Right. It's simple, yeah.
> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2299
> > <https://reviews.apache.org/r/15993/diff/6/?file=395989#file395989line2299>
> >
> > Not necessary to fix todo in this jira. But will be good an example where prefix could be longer than maxPrefixLength.
ok.
> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2318
> > <https://reviews.apache.org/r/15993/diff/6/?file=395989#file395989line2318>
> >
> > Looks like this method is no longer used anywhere. if so, can be removed.
Soley used in getHashedStatsPrefix(). I've inlined that.
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/#review30071
-----------------------------------------------------------
On Dec. 10, 2013, 12:42 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15993/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2013, 12:42 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-5936
> https://issues.apache.org/jira/browse/HIVE-5936
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
>
>
> Diffs
> -----
>
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java a9c3136
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java d0e94bf
> metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
> ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8
> ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6
> ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea4
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7
> ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java fa430eb
> ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648
> ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
> ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java fb5f50e
> ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 8c23b87
> ql/src/test/queries/clientpositive/stats_counter.q 20769e4
> ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
> ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
> ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607
> ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
> ql/src/test/results/clientpositive/truncate_column.q.out a247c4a
>
> Diff: https://reviews.apache.org/r/15993/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/#review30071
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
<https://reviews.apache.org/r/15993/#comment57612>
I don't think we need to call init() from task nodes. Semantics is : init() is called once from client before job is launched. Is there some reason to call init() here now?
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
<https://reviews.apache.org/r/15993/#comment57613>
This can simply be written as
String spSpec = conf.getStaticSpec();
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
<https://reviews.apache.org/r/15993/#comment57614>
Will be good to add a comment in javadoc about which strings are contained in different positions of array.
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
<https://reviews.apache.org/r/15993/#comment57615>
Nice work to batch together these metastore updates.
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
<https://reviews.apache.org/r/15993/#comment57616>
invalid semantics (todo) we should probably fix in seperate jira. There I think we need to make a distinction between insert overwrite and insert into cases, for earlier we can replace invalid stats with current value but for insert into they should remain invalid.
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
<https://reviews.apache.org/r/15993/#comment57617>
Same comment about needing init() here.
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
<https://reviews.apache.org/r/15993/#comment57618>
Not necessary to fix todo in this jira. But will be good an example where prefix could be longer than maxPrefixLength.
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
<https://reviews.apache.org/r/15993/#comment57619>
Looks like this method is no longer used anywhere. if so, can be removed.
- Ashutosh Chauhan
On Dec. 10, 2013, 12:42 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15993/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2013, 12:42 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-5936
> https://issues.apache.org/jira/browse/HIVE-5936
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
>
>
> Diffs
> -----
>
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java a9c3136
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java d0e94bf
> metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
> ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8
> ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80
> ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6
> ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea4
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7
> ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03
> ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java fa430eb
> ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648
> ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
> ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java fb5f50e
> ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 8c23b87
> ql/src/test/queries/clientpositive/stats_counter.q 20769e4
> ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
> ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
> ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607
> ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
> ql/src/test/results/clientpositive/truncate_column.q.out a247c4a
>
> Diff: https://reviews.apache.org/r/15993/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 16, 2013, 1:27 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
Fixed TestMTQueries fails (failing stat aggregation by trying concurrent derby connection)
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java a9c3136
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 378de03
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java c16e82d
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java 19f88ee
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/KeyVerifyingStatsAggregator.java 8fa5c3e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 3deed45
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 7c61c72
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 19f7d79
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java fdc0d1a
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 17e6aad
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ace1df9
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticException.java e3dca57
ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java fa430eb
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java fb5f50e
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 8c23b87
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out 9a6e38f
ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out 2ba99b1
ql/src/test/results/clientnegative/stats_publisher_error_2.q.out 5284672
ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out 5735c4f
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607
ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
ql/src/test/results/clientpositive/stats_publisher_error_1.q.out a122b83
ql/src/test/results/clientpositive/truncate_column.q.out a247c4a
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 10, 2013, 6:03 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java a9c3136
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java d0e94bf
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea4
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7
ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java fa430eb
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java fb5f50e
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 8c23b87
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607
ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
ql/src/test/results/clientpositive/truncate_column.q.out a247c4a
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 10, 2013, 12:42 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java a9c3136
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java d0e94bf
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea4
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7
ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java fa430eb
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java fb5f50e
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 8c23b87
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607
ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
ql/src/test/results/clientpositive/truncate_column.q.out a247c4a
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 9, 2013, 5:28 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
Minor refactorings to simplify things
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java d0e94bf
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 08b9e58
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java fa430eb
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 8c23b87
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
ql/src/test/results/clientpositive/stats_noscan_1.q.out 054ca38
ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 5, 2013, 5:04 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
fixed the case when maxLength - posfixLength < 0
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java e7453c7
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 142af10
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 8d895f4
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 91a9e6c
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 0268f98
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e9d9ee7
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 2cc2519
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
ql/src/test/results/clientpositive/stats_noscan_1.q.out 054ca38
ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 5, 2013, 3:37 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
1. Changed not to reset stats when "compute statistics noscan" is used
2. Removed assert in Utilities.getHashedStatsPrefix() which is added in previous diff. That makes some tests fail (stats19.q, etc.)
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java e7453c7
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 142af10
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 8d895f4
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 91a9e6c
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 0268f98
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e9d9ee7
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 2cc2519
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
ql/src/test/results/clientpositive/stats_noscan_1.q.out 054ca38
ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 15993: analyze command failing to collect stats with
counter mechanism
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
-----------------------------------------------------------
(Updated Dec. 4, 2013, 5:28 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936
Repository: hive-git
Description
-------
With counter mechanism, MR job is successful, but StatsTask on client fails with NPE
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2
ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 142af10
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 91a9e6c
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 7e701f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 0268f98
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e9d9ee7
ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 2cc2519
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0
ql/src/test/queries/clientpositive/stats_counter.q 20769e4
ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5
Diff: https://reviews.apache.org/r/15993/diff/
Testing
-------
Thanks,
Navis Ryu