You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2020/12/16 19:14:47 UTC

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString 

Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16884


Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................

IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

BufferedTupleStream::DebugString() iterate std::list<Page> that can
potentially grow very large. As consequent, the returned string can grow
large as well and cause a problem as previously happen in IMPALA-9851.
With this patch, BufferedTupleStream::DebugString() only include maximum
of 100 first pages of page list.

Testing:
- Add new be test SimpleNullStreamTest.ShortDebugString in
  buffered-tuple-stream-test.cc
- Pass core tests

Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
---
M be/src/runtime/buffered-tuple-stream-test.cc
M be/src/runtime/buffered-tuple-stream.cc
M be/src/runtime/buffered-tuple-stream.h
3 files changed, 71 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/16884/1
-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 2: Code-Review+2

(1 comment)

LGTM. Thanks for handling this!

http://gerrit.cloudera.org:8080/#/c/16884/2/be/src/runtime/buffered-tuple-stream.h
File be/src/runtime/buffered-tuple-stream.h:

http://gerrit.cloudera.org:8080/#/c/16884/2/be/src/runtime/buffered-tuple-stream.h@563
PS2, Line 563:   friend class ArrayTupleStreamTest_TestArrayDeepCopy_Test;
             :   friend class ArrayTupleStreamTest_TestComputeRowSize_Test;
             :   friend class MultiNullableTupleStreamTest_TestComputeRowSize_Test;
> I notice that since IMPALA-10337 changed BufferedTupleStream::ComputeRowSiz
Yeah, please delete them.



-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 02:31:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 04:57:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6803/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Dec 2020 05:11:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 2:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc
File be/src/runtime/buffered-tuple-stream-test.cc:

http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc@1472
PS1, Line 1472: SimpleTupleStreamTest
> nit: To be consistent with other tests, I think the label should be "Simple
Done


http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc@1485
PS1, Line 1485: bool b = stream.AddRow(batch->GetRow(j), &status);
> I think we can remove this TODO.
Done


http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc@1508
PS1, Line 1508: 
> Can we calculate 148 out based on the vars? I'm concerning it will introduc
It is not obvious for me to calculate it based on the vars.
So instead, I change it to look up stream.num_pages_ directly and add verification that num_pages_ > MAX_PAGE_ITER_DEBUG, given the workload.


http://gerrit.cloudera.org:8080/#/c/16884/2/be/src/runtime/buffered-tuple-stream.h
File be/src/runtime/buffered-tuple-stream.h:

http://gerrit.cloudera.org:8080/#/c/16884/2/be/src/runtime/buffered-tuple-stream.h@563
PS2, Line 563:   friend class ArrayTupleStreamTest_TestArrayDeepCopy_Test;
             :   friend class ArrayTupleStreamTest_TestComputeRowSize_Test;
             :   friend class MultiNullableTupleStreamTest_TestComputeRowSize_Test;
I notice that since IMPALA-10337 changed BufferedTupleStream::ComputeRowSize() from private to public, these lines are not required anymore.
Can I delete them along with this patch?


http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream.cc
File be/src/runtime/buffered-tuple-stream.cc:

http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream.cc@189
PS1, Line 189: \
> Comparing to the original codes, we should have "\n" here.
Done. Thanks for catching this!



-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Dec 2020 22:14:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16884

to look at the new patch set (#3).

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................

IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

BufferedTupleStream::DebugString() iterate std::list<Page> that can
potentially grow very large. As consequent, the returned string can grow
large as well and cause a problem as previously happen in IMPALA-9851.
With this patch, BufferedTupleStream::DebugString() only include maximum
of 100 first pages of page list.

Testing:
- Add new be test SimpleTupleStreamTest.ShortDebugString in
  buffered-tuple-stream-test.cc
- Pass core tests

Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
---
M be/src/runtime/buffered-tuple-stream-test.cc
M be/src/runtime/buffered-tuple-stream.cc
M be/src/runtime/buffered-tuple-stream.h
3 files changed, 73 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/16884/3
-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Dec 2020 10:42:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6800/


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 10:45:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 04:58:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16884/2/be/src/runtime/buffered-tuple-stream.h
File be/src/runtime/buffered-tuple-stream.h:

http://gerrit.cloudera.org:8080/#/c/16884/2/be/src/runtime/buffered-tuple-stream.h@563
PS2, Line 563:   friend class SimpleTupleStreamTest_ShortDebugString_Test;
             : 
             :   /// Runtime state instance used to check for cancellation. Not own
> Yeah, please delete them.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 03:26:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 4:

> Patch Set 4: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6800/

Hit by IMPALA-9550.
Ironically, I just close IMPALA-9550 yesterday as not reproducible. I will reopen the JIRA again.


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 15:46:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7891/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Dec 2020 22:12:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 5:

Patch set 5 is a rebase on top of IMPALA-9550 fix.
The flakiness should be gone now.


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Dec 2020 01:33:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7894/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 03:41:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16884

to look at the new patch set (#2).

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................

IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

BufferedTupleStream::DebugString() iterate std::list<Page> that can
potentially grow very large. As consequent, the returned string can grow
large as well and cause a problem as previously happen in IMPALA-9851.
With this patch, BufferedTupleStream::DebugString() only include maximum
of 100 first pages of page list.

Testing:
- Add new be test SimpleTupleStreamTest.ShortDebugString in
  buffered-tuple-stream-test.cc
- Pass core tests

Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
---
M be/src/runtime/buffered-tuple-stream-test.cc
M be/src/runtime/buffered-tuple-stream.cc
M be/src/runtime/buffered-tuple-stream.h
3 files changed, 73 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/16884/2
-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7868/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Dec 2020 19:36:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................

IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

BufferedTupleStream::DebugString() iterate std::list<Page> that can
potentially grow very large. As consequent, the returned string can grow
large as well and cause a problem as previously happen in IMPALA-9851.
With this patch, BufferedTupleStream::DebugString() only include maximum
of 100 first pages of page list.

Testing:
- Add new be test SimpleTupleStreamTest.ShortDebugString in
  buffered-tuple-stream-test.cc
- Pass core tests

Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Reviewed-on: http://gerrit.cloudera.org:8080/16884
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
M be/src/runtime/buffered-tuple-stream-test.cc
M be/src/runtime/buffered-tuple-stream.cc
M be/src/runtime/buffered-tuple-stream.h
3 files changed, 73 insertions(+), 7 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Quanlong Huang: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6805/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Dec 2020 13:21:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6800/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 04:58:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 1:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc
File be/src/runtime/buffered-tuple-stream-test.cc:

http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc@1472
PS1, Line 1472: SimpleTupleStreamTest
nit: To be consistent with other tests, I think the label should be "SimpleTupleStreamTest::ShortDebugString"


http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc@1485
PS1, Line 1485: // TODO: test that AddRow succeeds after freeing memory.
I think we can remove this TODO.


http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream-test.cc@1508
PS1, Line 1508: 148
Can we calculate 148 out based on the vars? I'm concerning it will introduce flakiness.


http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream.cc
File be/src/runtime/buffered-tuple-stream.cc:

http://gerrit.cloudera.org:8080/#/c/16884/1/be/src/runtime/buffered-tuple-stream.cc@189
PS1, Line 189:  
Comparing to the original codes, we should have "\n" here.



-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Dec 2020 08:45:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16884 )

Change subject: IMPALA-10374: Limit iteration at BufferedTupleStream::DebugString
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/16884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6626c8d54f35f303c01f85be1dd9aa54c8ad9a2d
Gerrit-Change-Number: 16884
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Dec 2020 13:19:53 +0000
Gerrit-HasComments: No