You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by jihoonson <gi...@git.apache.org> on 2015/07/24 06:39:19 UTC
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
GitHub user jihoonson opened a pull request:
https://github.com/apache/tajo/pull/651
TAJO-1300: Merge the index branch into the master branch
I think that ```index_support``` branch is ready to be merged into ```master```.
In this branch, I've developed Tajo to support index feature. Here are some highlight features.
* Support CREATE / DROP INDEX statement (TAJO-836, TAJO-837)
* Improve query planner to utilize index if possible (TAJO-838)
* Support index metadata backup and restore (TAJO-1301, TAJO-1302)
There remains some additional features to improve the support on index including https://issues.apache.org/jira/browse/TAJO-1164. I'll proceed them in https://issues.apache.org/jira/browse/TAJO-835.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jihoonson/tajo-2 TAJO-1300
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/651.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #651
----
commit 31d524c51cc29da1c90b2431279ca63f24a19172
Author: Jihoon Son <ji...@apache.org>
Date: 2014-07-11T02:47:19Z
TAJO-836: create index support. (jihoon)
commit 6d6b968f3259acd5d6c82f41ae0e6945c68daa28
Author: Jihoon Son <ji...@apache.org>
Date: 2014-07-21T02:44:57Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit bf5b43c40f10de2bd47277209a726cfe92776f7b
Author: Jihoon Son <ji...@apache.org>
Date: 2014-09-25T12:13:55Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
Conflicts:
tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanPreprocessor.java
tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
commit 3777f58251e61cd965b8bf5200f9a59921adfe3f
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-01T22:09:23Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit d1d4d25fb0d08f0846e758c43caf3e3f3b1bc543
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-02T00:40:58Z
tmp
commit 7f98fe78613ce6349c94dba559f7e4b4197b18e9
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-02T08:49:58Z
test success
commit 6ced5a10881e93fd09406cd4dba6dd88e54c09bd
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-02T14:50:27Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit e54dd8e014776f0ef58774013c06120cc01eeb54
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-02T14:52:57Z
Merge branch 'index_support' of github.com:jihoonson/tajo-2 into index_support
commit 375eb1ab2d457f577cb22a2239eec017fa7b6409
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-03T04:08:48Z
remove commented out codes
commit afc2c3eb3058a0473b7fb2e4808e5b0dbd71df4e
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-10T04:32:25Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
Conflicts:
tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/IndexDesc.java
commit daba9fdae460c163444f00dfc3a79d2d8d220aa7
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-12T07:51:05Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 27880386af0781e4cd6333e2b11769b04592192c
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-14T08:19:54Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit f98b78cf993ef9194ef494f669bec49d3d663ff4
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-14T08:20:34Z
Merge branch 'index_support' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit fbb95ec62fd52d648e60424e249ffc390ad1ee39
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-16T06:02:34Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 738c3a1de4d75702fb8e0c54de4c8f3af4b8f83b
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-17T01:48:16Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit cd613507907440acd652341138c6d76a1e9ae568
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-19T11:21:55Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 322da3885d7e806b875158725cc3885e39f60be2
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-19T11:22:08Z
Merge branch 'index_support' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 4fb0f80e297c91b4940c214219eff1aae3959804
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-23T11:37:13Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 4263b430063342bb28c887ad63df9bce2fa48fde
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-26T05:45:34Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
Conflicts:
tajo-plan/src/main/java/org/apache/tajo/plan/visitor/BasicLogicalPlanVisitor.java
tajo-plan/src/main/java/org/apache/tajo/plan/visitor/LogicalPlanVisitor.java
commit 561cae639bdd21f9a0387979c3b6bc22a55eb086
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-27T09:01:35Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit c669b62cd13a5635d4fa557efb5fc278567997a7
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-28T02:20:37Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 7970ac2d5cba8cff0cba361e70e827de125ab9d8
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-29T05:49:13Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 567b1c4832607383ccab1cc3bdd282f732434657
Author: Jihoon Son <ji...@apache.org>
Date: 2014-10-31T05:17:24Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 6a798c088c4528550442a05eb3a428b999ece6f0
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-02T10:58:43Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 70de1b443c2dd85b4ec87267c33c409c520b5854
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-04T12:31:35Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 0dc55a882c8922d0d36cdbb1174e162ec5d97030
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-09T09:03:57Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 62bd3a3401476ca88c45b67b8dd07d3f76b3cc8d
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-10T08:18:51Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit a5298fb216005ec30ca9774165132d0af92eaeb0
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-10T11:30:34Z
Merge branch 'master' of http://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 4a364cf08a22ab50eb50eb6b8ca373e2f6f0ce42
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-11T14:24:30Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
commit 02b827e2499036374be852c2d98526ae8f0f0d84
Author: Jihoon Son <ji...@apache.org>
Date: 2014-11-14T04:46:08Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into index_support
Conflicts:
tajo-catalog/tajo-catalog-server/src/main/resources/schemas/derby/indexes.sql
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124405331
You are right. I will book those issues in other Jiras.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124432337
After pulling latest codes, I found another error message on tsql as following:
```
2015-07-24 17:12:00,767 ERROR org.apache.tajo.catalog.CatalogServer:
Stack Trace:
java.lang.StackOverflowError
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:121)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
at org.apache.tajo.exception.TajoRuntimeException.<init>(TajoRuntimeException.java:43)
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:136)
```
This issue looks like an issue of the master branch without relevant to this PR. If you don't mind, could you recover relative codes? We need to book this issue with another JIRA issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124412695
Right, but it's not related to this issue.
Honestly, I've already created an issue for documentation (https://issues.apache.org/jira/browse/TAJO-1705).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-126167782
+1
The patch looks good to me. Finally, we can merge it to master branch. Thank you for your effort.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124347074
Hi @jihoonson
I found that TajoMaster printed a error message when executing *\d* command on tsql as follows.
# Tsql
```
default> \d nation;
table name: default.nation
table uri: hdfs://localhost:9010/tpch/nation
store type: TEXT
number of rows: unknown
volume: 2.2 kB
Options:
'textfile.delimiter'='|'
'text.delimiter'='|'
schema:
n_nationkey INT8
n_name TEXT
n_regionkey INT8
n_comment TEXT
```
# TajoMaster
```
2015-07-24 15:29:08,821 ERROR org.apache.tajo.catalog.CatalogServer:
Stack Trace:
org.apache.tajo.exception.TajoInternalError: internal error: no error message for UNDEFINED_INDEX
at org.apache.tajo.exception.ErrorMessages.getMessage(ErrorMessages.java:119)
at org.apache.tajo.exception.ReturnStateUtil.returnError(ReturnStateUtil.java:79)
at org.apache.tajo.exception.ReturnStateUtil.errUndefinedIndex(ReturnStateUtil.java:176)
at org.apache.tajo.catalog.CatalogServer$CatalogProtocolHandler.existIndexesByTable(CatalogServer.java:1136)
at org.apache.tajo.catalog.AbstractCatalogClient.existIndexesByTable(AbstractCatalogClient.java:571)
at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.existIndexesForTable(TajoMasterClientService.java:1066)
at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:711)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124357951
@blrunner, thanks. This is a bug while applying the recent changes on error propagation. Since this exception is not exposed to users, I couldn't recognized it.
Anyway, I'll fix soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124414320
OK. I got it. :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124449676
That error is already fixed at https://github.com/apache/tajo/commit/98b841dd543866997c1d2e806dc0d3a40cf6beee.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124788814
I removed existing codes and got latest source codes of this PR. *mvn clean install -Pparallel-test -DLOG_LEVEL=WARN -Dmaven.fork.count=2* have been finished successfully. But above situation is still reproduced. Could you check it again or recover it to as-found condition?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124400523
I found that relative index functions of HiveCatalogStore just throws UnsupportedOperationException. Hive already provides relative index functions to client, for examples, createIndex, dropIndex, getIndex, alterIndex. Even if not in right now, you need to consider HiveCatalogStore implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-126180130
I'm very happy with that I finally committed this patch. Thanks guys!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/651
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124374074
OK, No problem. :-)
And if there is a message when executing *explain* command on tsql, this patch would be more good. Currently, I can't know whether index scan is applied or not before executing select statement as following:
```
default> create external table lineitem (L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY bigint, L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text, L_SHIPDATE text, L_COMMITDATE text, L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text) using text with ('textfile.delimiter'='|') location 'hdfs://localhost:9010/tpch/lineitem';
OK
default> create index l_orderkey_idx on lineitem (l_orderkey);
default> \set INDEX_ENABLED true
default> explain select count(*) from lineitem where l_orderkey = 1;
explain
-------------------------------
GROUP_BY(3)()
=> exprs: (count())
=> target list: ?count (INT8)
=> out schema:{(1) ?count (INT8)}
=> in schema:{(0) }
(5 rows, 0.034 sec, 0 B selected)
default> select count(*) from lineitem where l_orderkey = 1;
Progress: 100%, response time: 0.581 sec
?count
-------------------------------
6
(1 rows, 0.581 sec, 2 B selected)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124410378
I found that index scan just applied using an equals sign (=) in a predicate. If you add valid predicate operators for index scan and a few examples to document, it would be more helpful to users. :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124796163
+1
Thanks @jihoonson and I found that it worked successfully.
Ship it. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1300: Merge the index branch into the mast...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/651#issuecomment-124793057
Thank. There is another problem at TajoDump. I've fixed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---