You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Daniel Becker (Code Review)" <ge...@cloudera.org> on 2020/06/22 10:54:07 UTC

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14735

Change subject: POC: Remote codegen (Milestone 1)
......................................................................

POC: Remote codegen (Milestone 1)

The Impala executor does everything as usual until the point when it
would optimize and compile the LLVM module. Instead, it sends the LLVM
bitcode to the compile server which optimizes it and sends back an
object file (handled in memory, not written to disk). The executor
blocks until it receives the compiled object file.

Note that the customization of the LLVM module (replacing function
calls) is still done by the executor.

This stage should not be used in production as it actually slows down
the code generation because it is done remotely but synchronously.

The server is a python script that calls llc to compile the code.
It will later be replaced probably by a special impalad that uses the
LLVM API to compile the code.

Communication is not secure.

Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
---
A be/src/codegen/LLVMSmallVectorMemoryBuffer.h
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
A be/src/codegen/simple_message.h
M testdata/bin/kill-all.sh
M testdata/bin/run-all.sh
A testdata/bin/server.py
7 files changed, 499 insertions(+), 90 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/14735/4
--
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 8:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/6570/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 8
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jul 2020 08:47:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.

Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................

POC: Remote codegen (Milestone 1)

Optional remote codegen, set by a query option. Can be used together
with async.

Note that the customization of the LLVM module (replacing function
calls) is still done by the executor.

The server is a python script that calls llc to compile the code.
It will later be replaced probably by a special impalad that uses the
LLVM API to compile the code.

Communication is not secure.

TODO: Now both async and remote are switched ON for testing, these
should be disabled by default.

Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
---
A be/src/codegen/LLVMSmallVectorMemoryBuffer.h
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
A be/src/codegen/simple_message.h
M be/src/runtime/fragment-state.cc
M be/src/service/fe-support.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M testdata/bin/kill-all.sh
M testdata/bin/run-all.sh
A testdata/bin/server.py
13 files changed, 617 insertions(+), 62 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/14735/6
--
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 5:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/6512/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2020 14:16:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 4:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py
File testdata/bin/server.py:

http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@12
PS4, Line 12: def save_log_file(contents):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@17
PS4, Line 17:  
flake8: E261 at least two spaces before inline comment


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@23
PS4, Line 23: def bc_to_ll(bc):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@41
PS4, Line 41: def has_non64bit_reloc(obj_file):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@67
PS4, Line 67: def compile_program(data):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@79
PS4, Line 79: [
flake8: E126 continuation line over-indented for hanging indent


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@106
PS4, Line 106: def to_message(data):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@111
PS4, Line 111: class MyTCPHandler(SocketServer.BaseRequestHandler):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@157
PS4, Line 157: class Server(SocketServer.ForkingMixIn, SocketServer.TCPServer):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@160
PS4, Line 160: def server_main():
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@171
PS4, Line 171: def client_main():
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/14735/4/testdata/bin/server.py@181
PS4, Line 181: if __name__ == "__main__":
flake8: E305 expected 2 blank lines after class or function definition, found 1



-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2020 10:54:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.

Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 7:

Use a Thrift server for the compile server.


-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 11 Jul 2020 15:18:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 7:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/6565/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 11 Jul 2020 15:40:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14735/7/be/src/codegen/llvm-codegen.cc
File be/src/codegen/llvm-codegen.cc:

http://gerrit.cloudera.org:8080/#/c/14735/7/be/src/codegen/llvm-codegen.cc@1152
PS7, Line 1152: Status try_compile_remotely(const TRemoteCompileMsg& llvm_module, TRemoteCompileMsg* obj_file_msg) {
line too long (100 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 11 Jul 2020 15:18:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.

Daniel Becker has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................

POC: Remote codegen (Milestone 1)

Optional remote codegen, set by a query option. Can be used together
with async.

The Impala executor does everything as usual until the point when it
would optimize and compile the LLVM module. Instead, it sends the LLVM
bitcode to the compile server which optimizes it and sends back an
object file (handled in memory, not written to disk). The executor
blocks until it receives the compiled object file.

Note that the customization of the LLVM module (replacing function
calls) is still done by the executor.

The server is a python script that calls llc to compile the code.
It will later be replaced probably by a special impalad that uses the
LLVM API to compile the code.

Communication is not secure.

Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
---
A be/src/codegen/LLVMSmallVectorMemoryBuffer.h
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
A be/src/codegen/simple_message.h
M be/src/runtime/fragment-state.cc
M be/src/service/fe-support.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M testdata/bin/kill-all.sh
M testdata/bin/run-all.sh
A testdata/bin/server.py
13 files changed, 611 insertions(+), 60 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/14735/5
-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/14735/5/be/src/codegen/llvm-codegen.h
File be/src/codegen/llvm-codegen.h:

http://gerrit.cloudera.org:8080/#/c/14735/5/be/src/codegen/llvm-codegen.h@346
PS5, Line 346:   /// If 'async' is true, start executing 'FinalizeModule' in a separate thread and return.
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/14735/5/be/src/service/fe-support.cc
File be/src/service/fe-support.cc:

http://gerrit.cloudera.org:8080/#/c/14735/5/be/src/service/fe-support.cc@242
PS5, Line 242:     status = codegen->FinalizeModule(false, false, nullptr); /// TODO: Check whether the arguments are ok.
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/14735/5/testdata/bin/server.py
File testdata/bin/server.py:

http://gerrit.cloudera.org:8080/#/c/14735/5/testdata/bin/server.py@207
PS5, Line 207: if __name__ == "__main__":
flake8: E305 expected 2 blank lines after class or function definition, found 1



-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2020 13:56:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 4:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/6387/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2020 11:24:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.

Daniel Becker has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................

POC: Remote codegen (Milestone 1)

Optional remote codegen, set by a query option. Can be used together
with async.

The Impala executor does everything as usual until the point when it
would optimize and compile the LLVM module. Instead, it sends the LLVM
bitcode to the compile server which optimizes it and sends back an
object file (handled in memory, not written to disk). The executor
blocks (in synchronous mode) until it receives the compiled object file.

Note that the customization of the LLVM module (replacing function
calls) is still done by the executor.

The server is a python script that calls llc to compile the code.
It will later be replaced probably by a special impalad that uses the
LLVM API to compile the code.

Communication is not secure.

TODO: Now both async and remote are switched ON for testing, these
should be disabled by default.

Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
---
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/runtime/fragment-state.cc
M be/src/service/fe-support.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CMakeLists.txt
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
A common/thrift/RemoteCompile.thrift
M testdata/bin/kill-all.sh
M testdata/bin/run-all.sh
A testdata/bin/server.py
14 files changed, 486 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/14735/7
-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 7
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.

Daniel Becker has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................

POC: Remote codegen (Milestone 1)

Optional remote codegen, set by a query option. Can be used together
with async.

The Impala executor does everything as usual until the point when it
would optimize and compile the LLVM module. Instead, it sends the LLVM
bitcode to the compile server which optimizes it and sends back an
object file (handled in memory, not written to disk). The executor
blocks (in synchronous mode) until it receives the compiled object file.

Note that the customization of the LLVM module (replacing function
calls) is still done by the executor.

The server is a python script that calls llc to compile the code.
It will later be replaced probably by a special impalad that uses the
LLVM API to compile the code.

Communication is not secure.

TODO: Now both async and remote are switched ON for testing, these
should be disabled by default.

Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
---
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/runtime/fragment-state.cc
M be/src/service/fe-support.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/CMakeLists.txt
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
A common/thrift/RemoteCompile.thrift
M testdata/bin/kill-all.sh
M testdata/bin/run-all.sh
A testdata/bin/server.py
14 files changed, 487 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/14735/8
-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 8
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] POC: Remote codegen (Milestone 1)

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14735 )

Change subject: POC: Remote codegen (Milestone 1)
......................................................................


Patch Set 6:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/6528/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/14735
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I375d969ceb31b3ee234d6ca56a77d508ae43bb0f
Gerrit-Change-Number: 14735
Gerrit-PatchSet: 6
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Jul 2020 10:36:09 +0000
Gerrit-HasComments: No