[jira] [Commented] (TAJO-2007) By default, Optimizer should use the table volume in TableStat.

["Hudson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121930#comment-15121930 ] 

Hudson commented on TAJO-2007:
------------------------------

SUCCESS: Integrated in Tajo-0.11.1-build #164 (See [https://builds.apache.org/job/Tajo-0.11.1-build/164/])
TAJO-2007: By default, Optimizer should use the table volume in (hyunsik: rev 0c9b6a6e968bd759f7a38da9a5f62810ea1501b7)
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin3.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual4.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testJoinWithMultipleJoinTypes.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testFullOuterJoin1.Sort.plan
* tajo-core-tests/src/test/resources/queries/TestTablePartitions/case3.sql
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testMultipleBroadcastDataFileWithZeroLength2.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithConstantExpr1.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinAsterisk.Hash.plan
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testComplexJoinsWithCaseWhen.Sort.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestHashSemiJoinExec.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithThetaJoinConditionInWhere.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinWithMultipleJoinQual1.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testWhereClauseJoin6.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinAsterisk.Sort_NoBroadcast.plan
* tajo-plan/src/main/java/org/apache/tajo/plan/LogicalOptimizer.java
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testComplexJoinsWithCaseWhen.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual4.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testTPCHQ2Join.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testFullOuterJoin1.Hash.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestUnionQuery.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestSortIntersectExec.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testWhereClauseJoin5.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual3.result
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin3.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithThetaJoinConditionInWhere.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinWithSubQuery/testLeftOuterJoinWithEmptySubquery1.1.Hash.plan
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession3.result
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithNull1.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual2.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithConstantExpr1.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithConstantExpr5.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin3.Sort_NoBroadcast.plan
* tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/LogicalPlanRewriteRuleContext.java
* tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterTask.java
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testRightOuterJoin1.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin3.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testBroadcastTwoPartJoin.Hash.plan
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession6.result
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSetSessionQuery.java
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testInnerAndOuterWithEmpty.1.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testFullOuterJoin1.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithConstantExpr4.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual4.Hash.plan
* tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/BaseLogicalPlanRewriteRuleProvider.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin1.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual3.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithConstantExpr4.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithNull3.Hash.plan
* tajo-core-tests/src/test/java/org/apache/tajo/querymaster/TestKillQuery.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestJoinOrderAlgorithm.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestMultipleJoinTypes.java
* tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testBroadcastTwoPartJoin.Sort.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestCrossJoin.java
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testInnerAndOuterWithEmpty.1.Hash.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestOuterJoinQuery.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual2.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testTPCHQ2Join.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithEmptyTable4.Sort.plan
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession4.result
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinOnMultipleDatabases.Hash_NoBroadcast.plan
* tajo-common/src/main/java/org/apache/tajo/validation/MinValidator.java
* tajo-common/src/main/java/org/apache/tajo/SessionVars.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testTPCHQ2Join.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinAsterisk.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testBroadcastSubquery2.Sort.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java
* tajo-core-tests/src/test/java/org/apache/tajo/master/TestExecutionBlockCursor.java
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testFullOuterJoin1.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestCrossJoin/testCrossJoinOfOneSmallTableAndJoin.1.Hash.plan
* tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinOnMultipleDatabases.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithNull1.Sort.plan
* tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/BaseLogicalPlanRewriteEngine.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testBroadcastTwoPartJoin.Hash_NoBroadcast.plan
* tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/TableStatUpdateRewriter.java
* tajo-cluster-tests/src/test/java/org/apache/tajo/TajoTestingCluster.java
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testComplexJoinsWithCaseWhen2.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testMultipleBroadcastDataFileWithZeroLength2.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinWithMultipleJoinQual1.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testWhereClauseJoin6.Sort.plan
* tajo-core/src/main/java/org/apache/tajo/parser/sql/SQLAnalyzer.java
* tajo-core-tests/src/test/resources/results/TestTajoCli/testHelpSessionVars.result
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithConstantExpr5.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinAsterisk.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testWhereClauseJoin6.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual4.Sort.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestInnerJoinWithSubQuery.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testWhereClauseJoin5.Hash.plan
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession5.result
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestInnerJoinQuery.java
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/TestLogicalOptimizer.java
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession1.result
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithNull3.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithNull2.Hash.plan
* tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual2.Sort.plan
* tajo-core-tests/src/test/resources/results/TestTPCH/testQ2FourJoins.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinWithMultipleJoinQual1.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinWithMultipleJoinQual1.Sort.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual2.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testJoinWithMultipleJoinTypes.Hash.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
* CHANGES
* tajo-core-tests/src/test/resources/results/TestMultipleJoinTypes/testComplexJoinsWithCaseWhen2.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin1.Sort.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestHashAntiJoinExec.java
* tajo-storage/tajo-storage-pgsql/src/test/java/org/apache/tajo/storage/pgsql/TestPgSQLQueryTests.java
* tajo-core-tests/src/test/resources/results/TestOuterJoinWithSubQuery/testLeftOuterJoinWithEmptySubquery1.1.Sort.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinWithSubQuery/testLeftOuterJoinWithConstantExpr2.Sort.plan
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession7.result
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testBroadcastTwoPartJoin.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithEmptyTable4.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin2.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testRightOuterJoin1.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testBroadcastSubquery2.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual3.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinWithSubQuery/testLeftOuterJoinWithConstantExpr2.Hash.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual3.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testWhereClauseJoin6.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestSQLAnalyzer/setsession2.result
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testTPCHQ2Join.Hash_NoBroadcast.plan
* tajo-core-tests/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testJoinWithMultipleJoinQual3.Hash.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoin2.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testBroadcastSubquery.Sort.plan
* tajo-core-tests/src/test/resources/results/TestCrossJoin/testCrossJoinOfOneSmallTableAndJoin.1.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinWithSubQuery/testBroadcastSubquery.Hash.plan
* tajo-core-tests/src/test/resources/results/TestTPCH/testTPCHQ5.plan
* tajo-core-tests/src/test/resources/results/TestOuterJoinQuery/testLeftOuterJoinWithNull2.Sort.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinOnMultipleDatabases.Sort_NoBroadcast.plan
* tajo-core-tests/src/test/resources/results/TestInnerJoinQuery/testJoinOnMultipleDatabases.Hash.plan


> By default, Optimizer should use the table volume in TableStat.
> ---------------------------------------------------------------
>
>                 Key: TAJO-2007
>                 URL: https://issues.apache.org/jira/browse/TAJO-2007
>             Project: Tajo
>          Issue Type: Improvement
>          Components: Planner/Optimizer
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.12.0, 0.11.1
>
>
> Currently, the optimizer by default gets table volumes through storage manager and employ them for join optimization. But, in some cases, it causes performance degradation because aggregating all file volumes is not cheap in large partitioned tables on S3 or HDFS.
> So, this patch improves TableStatUpdateRewriter to use table volumes of TableStat by default, and it also adds a session variable 'USE_TABLE_VOLUME' to allow the optimizer to use the table volume through storage handler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)