You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by haohui <gi...@git.apache.org> on 2015/10/13 00:49:07 UTC
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
GitHub user haohui opened a pull request:
https://github.com/apache/storm/pull/796
STORM-1105. Compile the logical plans into LLVM functions.
This PR compiles three logical operators, tablescan, filter and projection to LLVM functions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/haohui/storm STORM-1105
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/796.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #796
----
commit 509ee73194077325ce7aab18970a13bedf33fdfa
Author: Haohui Mai <wh...@apache.org>
Date: 2015-10-12T22:41:55Z
STORM-1105. Compile the logical plans into LLVM functions.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
Posted by haohui <gi...@git.apache.org>.
Github user haohui closed the pull request at:
https://github.com/apache/storm/pull/796
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on the pull request:
https://github.com/apache/storm/pull/796#issuecomment-148294057
Thanks, I get it now. I think it would be good to leave some hooks to plugin other approaches to bytecode generation in future if needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the pull request:
https://github.com/apache/storm/pull/796#issuecomment-148433347
Thanks for the feedback. The architecture will be flexible to allow pluggable code generation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
Posted by haohui <gi...@git.apache.org>.
Github user haohui commented on the pull request:
https://github.com/apache/storm/pull/796#issuecomment-148118238
The main motivation is performance -- The code that generated by Calcite has a lot of down casts which can potentially have a visible impact, as the targeted throughput is millions of messages per seconds.
Bringing LLVM allows StormSQL to leverage the mature optimization and opens up further opportunities including automatic vectorization, etc.
Note that LLVM is used only for optimization -- the next step is to translate LLVM IR back to Java Bytecode (STORM-1097). The approach is quite similar to using Janino to generate Java bytecode. There is no native components in the worker node.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on the pull request:
https://github.com/apache/storm/pull/796#issuecomment-147949951
Curious to know why LLVM with C++ was chosen ? Calcite spark adapter appears to be using Janino to compile the generated java code. https://github.com/apache/incubator-calcite/tree/master/spark
Just a thought if we could have avoided native code by doing something similar, however I am not fully aware of the use cases that we are trying to address so I may be wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-1105. Compile the logical plans into LLV...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/796#issuecomment-147737771
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---