You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by hyunsik <gi...@git.apache.org> on 2014/08/11 08:12:58 UTC
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
GitHub user hyunsik opened a pull request:
https://github.com/apache/tajo/pull/113
TAJO-906: Runtime code generation for evaluating expression trees.
This is still an ongoing work. I share it for reviewing an overall approach.
Note that this patch includes some third party library because the version of ow2.asm is conflict to Hadoop's one. So, we don't need to review ow2.asm.
Also, this patch still does not pass one unit test related to IntervalDatum. I'll fix it soon.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hyunsik/tajo TAJO-906
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/113.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #113
----
commit b922a35a0f76e4ebc172d51418ed84e92837054f
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-02T13:18:44Z
added some sample codes.
commit 6dd5a2769d6b8be28b328647844401ddbca77c07
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-14T01:06:14Z
add some base codegen.
commit f735f6d35071878368f3c81591058dfe01c904a9
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-14T01:06:58Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit d9bc2f00be3b5b5d713d7c2a6691b9fffb024824
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-14T17:55:44Z
add arithmetic generation code.
commit 9146667dcc452156ed18ad4c46a1ea45068a9a75
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-15T05:11:42Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit a83355b12be2d6f5157156147e8634b545443aa8
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-16T15:02:58Z
Removed duplicated InvalidCastException and Refactored InvalidCastException and InvalidOperation.
commit 32dbcde96c44db6c16c859877dfd855f8cb6ef3c
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-16T15:03:31Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 72fbf9c86e969f5a2c1a3dd423638d3ac816fa1d
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-17T00:38:52Z
Refactor EvalNode to be one of UnaryEval, BinaryEval, or subclass of EvalNode.
commit 1dab2aa71a1889a557542e354f810c9ad2c87908
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-17T01:38:52Z
fill visitConst()
commit ae74d2bb2d747a6aa69fe11e7d98906faa047763
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-18T15:06:08Z
Add visitField and improved ExprBaseTest to use CodeGen.
commit 8044fd9ee624c9fa3d72d64e3f106e772950825c
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-18T16:07:25Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
Conflicts:
tajo-common/src/main/java/org/apache/tajo/exception/InvalidCastException.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/InvalidCastException.java
tajo-core/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
tajo-core/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java
tajo-core/tajo-core-backend/pom.xml
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/InvalidCastException.java
commit 4e1e55902e06690d2db82d91bc7c1be2fa68d758
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-19T06:08:40Z
Refactoring some method and simplified arithmetic operators.
commit af8ca5a398078f621ab7bb97b9dd187261e48469
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-19T06:09:02Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 741b1b67f833f88f4f5fd7d94140adc2fe1d350f
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-21T19:20:24Z
add comparison code generator.
commit fe4fca74007e7b7085034d5c791883604653a221
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-22T16:51:28Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
Conflicts:
tajo-core/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/NotEval.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/SignedEval.java
commit 840eaf29f7da323a3ba88f27333cd15e56aadf33
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-22T16:51:37Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 06d08c22b75438ddb2a2bf97c5dc37c8492fcc90
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-25T20:24:58Z
passed all tests.
commit 493aa9f47e703cbb1b758503bc31606fc0fa4fd8
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-26T17:50:00Z
Improved many things.
commit a467f85b5a1bb94780782e37ab3a517285f9f1df
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-26T17:50:10Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 924984533c920a35df0d328cd1b3e86722f4180d
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-26T20:52:37Z
added unit tests and improved null handling for all operations.
commit 6255737530ef3f887f8eda22726e567cd713bafa
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-27T19:17:57Z
improved implicit type conversion.
commit 72bc4b06b67522e15050c29cfdc8ff3d44a5c238
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-27T19:18:14Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 54d1a7acb798a5c0134a0c249f8ba7acd23a9365
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-28T14:13:11Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 61aca902f291d3184b3320828bdf36d331615fee
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-28T14:55:58Z
all unit tests are passed.
commit 00a1498e99a499f9fa18bcc0c86e20c34f6680ec
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-04-29T01:55:06Z
Moved more test from codegen.
commit a1674079f9e7cd17059e745a8dbcad511a5c23b8
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-05-04T17:49:58Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
Conflicts:
tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
tajo-common/src/main/java/org/apache/tajo/datum/BooleanDatum.java
tajo-common/src/main/java/org/apache/tajo/datum/exception/InvalidCastException.java
tajo-common/src/main/java/org/apache/tajo/exception/InvalidCastException.java
tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
tajo-core/src/main/java/org/apache/tajo/engine/codegen/CodeGenException.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/AlgebraicUtil.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/EvalType.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/SimpleEvalNodeVisitor.java
tajo-core/src/main/java/org/apache/tajo/engine/eval/UnaryEval.java
tajo-core/src/main/java/org/apache/tajo/engine/planner/ExprAnnotator.java
tajo-core/src/test/java/example/Example.java
tajo-core/src/test/java/org/apache/tajo/engine/eval/TestSQLExpression.java
commit c3dbd41cba5156eaf1badca78a4419ad78a15d18
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-05-08T05:03:08Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 80a9566acb8bf2ef3624d31330b46070a8dd97cf
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-05-08T16:41:54Z
Add between clause.
commit 3618d7da5a9ed1d7ea75c6f13de26d43d6ab9fd3
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-05-08T16:42:09Z
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into CodeGen
commit 7baff45c551920fa2c2d26d2163687d936a11bba
Author: Hyunsik Choi <hy...@apache.org>
Date: 2014-05-08T16:51:08Z
Removed the case where the length of char type is 1.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-53154532
Hi @blrunner,
If CODEGEN session variable is enabled, all queries will work by using code generation. But, currently, we cannot expect its performance improvement. The code generation is designed to avoid Datum objects creations during any computations and reduce interpretation overheads like branches.
But, in order to keep the compatibility against our current Tuple and Datum mechanism, code generation feature still create lots of Datum objects. Currently, I'm working new tuple structure using direct memory, which uses a sequence of bytes as a row blocks instead of an array of Datum objects. After than, it will give big performance benefits.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-53152719
Thanks for the review. I've updated the follows according to ASF license policy.
* ASM is BSD license. So, according to [Modifications to NOTICE](http://www.apache.org/dev/licensing-howto.html#mod-notice), I didn't modify NOTICE file.
* Instead, I've added ASM license to only LICENSE file.
* Removed ASF standard license header from ASM source codes.
* Please refer to [Treatment of Third-Party Works](http://www.apache.org/legal/src-headers.html)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-53156149
+1
Thank you for your detailed comments.
I'm also looking forward for it to improve tajo performance. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-52966395
I've rebased and fixed many bugs. Even though there are some trivial issues to do, this patch is ready to be reviewed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-53153634
I checked on ASF license deletion from ASM source codes. And I double-checked tajo LICENSE file.
'mvn clean install -DCODEGEN=true' finished successfully. And then I found that CODEGEN variable on tsql as follows:
{code:xml}
default> \set CODEGEN true
default> \set
'CURRENT_DATABASE'='default'
'CODEGEN'='true'
'SESSION_ID'='f902f118-2649-46df-97ec-f15c2f0c75f5'
'USERNAME'='blrunner'
'SESSION_LAST_ACCESS_TIME'='1408801963706'
{code}
But I have a question about this patch. Is there any way of checking runtime code generation on local cluster?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by hyunsik <gi...@git.apache.org>.
Github user hyunsik commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-53052194
I've updated the patch. I've improved TajoTestingCluster to take system properties as follows:
```
mvn clean install -DCODEGEN=true
```
If ```CODEGEN``` is a session variable, it will be applied to QueryContext instance used in all unit tests. So, in order to test the code generation feature, you should give ```-DCODEGEN=true``` when you execute ```mvn install```. It can be used for other session variables too.
For test for real queries, you need to set a session variable CODEGEN as follows:
```
tajo> \set CODEGEN true
tajo>
tajo> SELECT .....
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/113
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-906: Runtime code generation for evaluatin...
Posted by blrunner <gi...@git.apache.org>.
Github user blrunner commented on the pull request:
https://github.com/apache/tajo/pull/113#issuecomment-53148187
Hi @hyunsik
This is a really great work. But I recommend for you to consider license.
If you add ASM project license to tajo license file and tajo notice file, it would not be a problem.
Cheers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---