You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/11/10 16:25:48 UTC

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20697


Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11542: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytesAllocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/SectionMemoryManager.cc
M be/src/codegen/SectionMemoryManager.h
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-cache.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/mcjit-mem-mgr.h
6 files changed, 19 insertions(+), 13 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20697

to look at the new patch set (#10).

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11805: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytes_allocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/mcjit-mem-mgr.h
M be/src/thirdparty/llvm/SectionMemoryManager.cpp
M be/src/thirdparty/llvm/SectionMemoryManager.h
5 files changed, 40 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/10
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 5: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 13 Nov 2023 14:21:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14480/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 20 Nov 2023 19:28:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 12: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 12
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Nov 2023 23:05:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20697

to look at the new patch set (#3).

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11542: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytesAllocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/SectionMemoryManager.cc
M be/src/codegen/SectionMemoryManager.h
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-cache.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/mcjit-mem-mgr.h
6 files changed, 19 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14457/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 16 Nov 2023 01:14:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Csaba Ringhofer, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20697

to look at the new patch set (#2).

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11542: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytesAllocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/SectionMemoryManager.cc
M be/src/codegen/SectionMemoryManager.h
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-cache.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/mcjit-mem-mgr.h
6 files changed, 19 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 9:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20697/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20697/5//COMMIT_MSG@7
PS5, Line 7: IMPALA-11805: Fix LLVM memory manager bytes allocated
> This should be part of IMPALA-11805.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 17 Nov 2023 17:15:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9968/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 12
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 29 Nov 2023 18:41:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20697

to look at the new patch set (#6).

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11542: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytes_allocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/mcjit-mem-mgr.h
M be/src/thirdparty/llvm/SectionMemoryManager.cpp
M be/src/thirdparty/llvm/SectionMemoryManager.h
4 files changed, 27 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/6
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 13: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 13
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 14:11:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 7:

I need to test this a bit more, had one intermittent failure of LlvmCodeGenCacheTest.BasicFunction fail and it's not clear why.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 15 Nov 2023 05:07:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 6:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14448/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 15 Nov 2023 05:35:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 13:

> Patch Set 13: Code-Review+1
> 
> LGTM!!
> Considering the rise in bytes_allocated after the change, have you run exhaustive tests to ensure the memory tracker can handle the increased demand?

Good idea, I'll do that.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 13
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 04 Dec 2023 18:46:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20697

to look at the new patch set (#7).

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11805: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytes_allocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/mcjit-mem-mgr.h
M be/src/thirdparty/llvm/SectionMemoryManager.cpp
M be/src/thirdparty/llvm/SectionMemoryManager.h
4 files changed, 27 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/7
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14417/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 16:53:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Yida Wu, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20697

to look at the new patch set (#9).

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11805: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytes_allocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
---
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/mcjit-mem-mgr.h
M be/src/thirdparty/llvm/SectionMemoryManager.cpp
M be/src/thirdparty/llvm/SectionMemoryManager.h
5 files changed, 38 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/20697/9
-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 13: Verified+1

> Patch Set 13:
> 
> > Patch Set 13: Code-Review+1
> > 
> > LGTM!!
> > Considering the rise in bytes_allocated after the change, have you run exhaustive tests to ensure the memory tracker can handle the increased demand?
> 
> Good idea, I'll do that.

Exhaustive run ran into IMPALA-12500, but otherwise passed.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 13
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 18:44:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 13: Code-Review+1

LGTM!!
Considering the rise in bytes_allocated after the change, have you run exhaustive tests to ensure the memory tracker can handle the increased demand?


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 13
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 04 Dec 2023 02:01:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................

IMPALA-11805: Fix LLVM memory manager bytes allocated

Previously the bytes_allocated function would undercount memory
allocated by the LLVM memory manager, as it allocates memory in full
pages but bytes_allocated only tracked the actual storage requested.
Implements bytes_allocated in the SectionMemoryManager to access the
actual storage size allocated.

Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Reviewed-on: http://gerrit.cloudera.org:8080/20697
Reviewed-by: Yida Wu <wy...@gmail.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/mcjit-mem-mgr.h
M be/src/thirdparty/llvm/SectionMemoryManager.cpp
M be/src/thirdparty/llvm/SectionMemoryManager.h
5 files changed, 40 insertions(+), 22 deletions(-)

Approvals:
  Yida Wu: Looks good to me, but someone else must approve
  Csaba Ringhofer: Looks good to me, approved
  Michael Smith: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 14
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 13:

Carrying Verified +1 through the rebase.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 13
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 18:44:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14416/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 16:53:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 3:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14418/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 17:10:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11542: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11542: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20697/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20697/5//COMMIT_MSG@7
PS5, Line 7: IMPALA-11542: Fix LLVM memory manager bytes allocated
This should be part of IMPALA-11805.



-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 15 Nov 2023 05:06:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11805: Fix LLVM memory manager bytes allocated

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20697 )

Change subject: IMPALA-11805: Fix LLVM memory manager bytes allocated
......................................................................


Patch Set 9:

This particular patch may become redundant with work Yida's doing to change how we cache codegen objects.


-- 
To view, visit http://gerrit.cloudera.org:8080/20697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a0193a1694db0718d75651dc2b2f9f92c0d6064
Gerrit-Change-Number: 20697
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 16 Nov 2023 00:53:33 +0000
Gerrit-HasComments: No