You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Yuqi Du (Code Review)" <ge...@cloudera.org> on 2022/02/18 13:36:50 UTC
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Yuqi Du has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18255
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will lead to
tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
wangningito has proposed a simple solution at https://gerrit.cloudera.org/c/16508/.
And I continue wangningito's work, and now use std::shared_ptr instead of scoped_refptr<Schema>,
because scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
27 files changed, 138 insertions(+), 139 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/1
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 1
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#6).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 152 insertions(+), 149 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/6
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 6
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#10).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/10
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 10
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#11).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/11
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 11
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 11: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG@23
PS11, Line 23: Conflicts:
: src/kudu/master/sys_catalog.cc
: src/kudu/tablet/tablet.cc
: src/kudu/tablet/tablet_metadata.cc
: src/kudu/tserver/tablet_service.cc
:
Remove these lines
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 11
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 08:33:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 8: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h
File src/kudu/tablet/tablet_metadata.h:
http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@379
PS8, Line 379: Set
nit: maybe remove the Set here, so it's just SwapSchemaUnlocked
http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@447
PS8, Line 447: // We don't use unique_ptr so that we can do an atomic swap.
nit: I'm not sure what this means. There doesn't seem to be much of a difference between swapping a unique_ptr vs a shared_ptr.
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 8
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Mar 2022 06:42:33 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#4).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
28 files changed, 143 insertions(+), 141 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/4
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 4
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#9).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/9
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 9
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 12: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 12
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 23:52:54 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#2).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
27 files changed, 137 insertions(+), 139 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/2
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 2
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#5).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
28 files changed, 143 insertions(+), 141 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/5
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 5
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 8:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc
File src/kudu/tablet/diskrowset.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc@633
PS3, Line 633: const SchemaPtr schema_ptr = rowset_metadata_->tablet_schema();
> Do we need to make a shared_ptr here? I suspect we do
yes. Done
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc
File src/kudu/tablet/ops/alter_schema_op.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc@107
PS3, Line 107: schema_ptr));
> Why not pass the shared ptr? Otherwise we end up creating a new shared ptr
Done
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc
File src/kudu/tablet/tablet.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: client_schema,
> Why there are two styles, one is declear a new variable, like 'SchemaPtr sc
Change it to: SchemaPtr schema_ptr = schema();
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: client_schema,
> +1, it's worth clarifying what the policy is for using shared_ptr vs raw po
yes. op_state's schema_at_decode_time is unsafe.
I would add a SchemaPtr to protect it.
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@1616
PS3, Line 1616: SchemaPtr schema = std::make_shared<Schema>(new_schema);
> How about use std::make_shared?
Done
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc
File src/kudu/tablet/tablet_metadata.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc@940
PS3, Line 940: }
:
> This may actually destruct the old schema, which could be expensive. Instea
good
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc@368
PS3, Line 368: tmeta->partition
> Don't we need to make a schema_ptr here?
Done
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 8
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Mar 2022 04:14:56 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 5:
(5 comments)
Sorry for the delay in review. Thanks for the contribution! Looking good so far :)
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc
File src/kudu/tablet/diskrowset.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc@633
PS3, Line 633: const Schema* schema = rowset_metadata_->tablet_schema().get();
Do we need to make a shared_ptr here? I suspect we do
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc
File src/kudu/tablet/ops/alter_schema_op.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc@107
PS3, Line 107: schema_ptr.get())
Why not pass the shared ptr? Otherwise we end up creating a new shared ptr in this call anyway.
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc
File src/kudu/tablet/tablet.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: client_schema,
> Why there are two styles, one is declear a new variable, like 'SchemaPtr sc
+1, it's worth clarifying what the policy is for using shared_ptr vs raw pointers.
For instances where the function expects a reference, I'm not sure it's safe to just pass the pointer, since it's possible the underlying schema is destroyed, since this call site doesn't keep a shared_ptr around. It'd be safer if we did something like:
SchemaPtr shared_schema = schema();
RowOperationsPBDecoder dec(&op_state->request()->row_operations(),
client_schema,
schema_ptr.get(),...
Or perhaps you're relying on the fact that we locked the schema at L538? If so, it's still worth in a comment, so other developers in this area of code better understand the nuances.
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc
File src/kudu/tablet/tablet_metadata.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc@940
PS3, Line 940: std::lock_guard<LockType> l(data_lock_);
: SetSchemaUnlocked(schema, version);
This may actually destruct the old schema, which could be expensive. Instead, maybe we should implement some SwapSchemaUnlocked() so the destruction doesn't happen while data_lock_ is held.
void TabletMetadata::SetSchema(SchemaPtr schema, uint32_t version) {
// In case this is the last reference to the schema, destruct the pointer
// outside the lock.
SchemaPtr old_schema;
{
lock_guard<LockType> l(data_lock_);
SwapSchemaUnlocked(std::move(schema), version, &old_schema);
}
}
void TabletMetadata::SwapSchemaUnlocked(SchemaPtr schema, uint32_t version, SchemaPtr* old_schema) {
*old_schema = std::move(schema_);
schema_ = std::move(schema);
schema_version_ = version;
}
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc@368
PS3, Line 368: *tmeta->schema()
Don't we need to make a schema_ptr here?
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 5
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Feb 2022 03:09:18 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#7).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 150 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/7
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 7
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 12:
(1 comment)
> Patch Set 11: Code-Review+1
>
> (1 comment)
http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG@23
PS11, Line 23: Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
:
:
:
:
:
> Remove these lines
Done
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 12
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 11:08:30 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#12).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/12
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 12
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Reviewed-on: http://gerrit.cloudera.org:8080/18255
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 13
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#3).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
27 files changed, 137 insertions(+), 139 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/3
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 3
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 3:
(2 comments)
Good work!
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc
File src/kudu/tablet/tablet.cc:
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: schema().get()
Why there are two styles, one is declear a new variable, like 'SchemaPtr schema_ptr = schema()', while another use 'schema().get()' directly?
http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@1616
PS3, Line 1616: SchemaPtr schema(new Schema(new_schema));
How about use std::make_shared?
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 3
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 11:59:45 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 13:
Thanks for the contribution!
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 13
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 00:17:02 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18255
to look at the new patch set (#8).
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.
Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.
The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/
Conflicts:
src/kudu/master/sys_catalog.cc
src/kudu/tablet/tablet.cc
src/kudu/tablet/tablet_metadata.cc
src/kudu/tserver/tablet_service.cc
Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 163 insertions(+), 150 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/8
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 8
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr
Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )
Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h
File src/kudu/tablet/tablet_metadata.h:
http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@379
PS8, Line 379: Sch
> nit: maybe remove the Set here, so it's just SwapSchemaUnlocked
Done
http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@447
PS8, Line 447: SchemaPtr schema_;
> nit: I'm not sure what this means. There doesn't seem to be much of a diff
This is an old comment. I will remove it.
--
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 9
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Mar 2022 14:20:09 +0000
Gerrit-HasComments: Yes