You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Yuqi Du (Code Review)" <ge...@cloudera.org> on 2022/02/18 13:36:50 UTC

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Yuqi Du has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18255


Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will lead to
tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

wangningito has proposed a simple solution at https://gerrit.cloudera.org/c/16508/.
And I continue wangningito's work, and now use std::shared_ptr instead of scoped_refptr<Schema>,
because scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
27 files changed, 138 insertions(+), 139 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 1
Gerrit-Owner: Yuqi Du <sh...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#6).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 152 insertions(+), 149 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/6
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 6
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#10).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/10
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 10
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#11).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/11
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 11
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 11: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG@23
PS11, Line 23: Conflicts:
             : 	src/kudu/master/sys_catalog.cc
             : 	src/kudu/tablet/tablet.cc
             : 	src/kudu/tablet/tablet_metadata.cc
             : 	src/kudu/tserver/tablet_service.cc
             : 
Remove these lines



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 11
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 08:33:16 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 8: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h
File src/kudu/tablet/tablet_metadata.h:

http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@379
PS8, Line 379: Set
nit: maybe remove the Set here, so it's just SwapSchemaUnlocked


http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@447
PS8, Line 447:   // We don't use unique_ptr so that we can do an atomic swap.
nit: I'm not sure what this  means. There doesn't seem to be much of a difference between swapping a unique_ptr vs a shared_ptr.



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 8
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Mar 2022 06:42:33 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#4).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
28 files changed, 143 insertions(+), 141 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 4
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#9).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/9
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 9
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 12: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 12
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 23:52:54 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#2).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
27 files changed, 137 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 2
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#5).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
28 files changed, 143 insertions(+), 141 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 5
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 8:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc
File src/kudu/tablet/diskrowset.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc@633
PS3, Line 633:   const SchemaPtr schema_ptr = rowset_metadata_->tablet_schema();
> Do we need to make a shared_ptr here? I suspect we do
yes. Done


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc
File src/kudu/tablet/ops/alter_schema_op.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc@107
PS3, Line 107: schema_ptr));
> Why not pass the shared ptr? Otherwise we end up creating a new shared ptr 
Done


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc
File src/kudu/tablet/tablet.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: client_schema,
> Why there are two styles, one is declear a new variable, like 'SchemaPtr sc
Change it to: SchemaPtr schema_ptr = schema();


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: client_schema,
> +1, it's worth clarifying what the policy is for using shared_ptr vs raw po
yes. op_state's schema_at_decode_time is unsafe.  

I would add a SchemaPtr to protect it.


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@1616
PS3, Line 1616:   SchemaPtr schema = std::make_shared<Schema>(new_schema);
> How about use std::make_shared?
Done


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc
File src/kudu/tablet/tablet_metadata.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc@940
PS3, Line 940: }
             : 
> This may actually destruct the old schema, which could be expensive. Instea
good


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc@368
PS3, Line 368: tmeta->partition
> Don't we need to make a schema_ptr here?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 8
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Mar 2022 04:14:56 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 5:

(5 comments)

Sorry for the delay in review. Thanks for the contribution! Looking good so far :)

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc
File src/kudu/tablet/diskrowset.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/diskrowset.cc@633
PS3, Line 633:   const Schema* schema = rowset_metadata_->tablet_schema().get();
Do we need to make a shared_ptr here? I suspect we do


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc
File src/kudu/tablet/ops/alter_schema_op.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/ops/alter_schema_op.cc@107
PS3, Line 107: schema_ptr.get())
Why not pass the shared ptr? Otherwise we end up creating a new shared ptr in this call anyway.


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc
File src/kudu/tablet/tablet.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: client_schema,
> Why there are two styles, one is declear a new variable, like 'SchemaPtr sc
+1, it's worth clarifying what the policy is for using shared_ptr vs raw pointers.

For instances where the function expects a reference, I'm not sure it's safe to just pass the pointer, since it's possible the underlying schema is destroyed, since this call site doesn't keep a shared_ptr around. It'd be safer if we did something like:

SchemaPtr shared_schema = schema();
RowOperationsPBDecoder dec(&op_state->request()->row_operations(),
                             client_schema,
                             schema_ptr.get(),...

Or perhaps you're relying on the fact that we locked the schema at L538? If so, it's still worth in a comment, so other developers in this area of code better understand the nuances.


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc
File src/kudu/tablet/tablet_metadata.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet_metadata.cc@940
PS3, Line 940:   std::lock_guard<LockType> l(data_lock_);
             :   SetSchemaUnlocked(schema, version);
This may actually destruct the old schema, which could be expensive. Instead, maybe we should implement some SwapSchemaUnlocked() so the destruction doesn't happen while data_lock_ is held.

void TabletMetadata::SetSchema(SchemaPtr schema, uint32_t version) {
  // In case this is the last reference to the schema, destruct the pointer
  // outside the lock.
  SchemaPtr old_schema;
  {
    lock_guard<LockType> l(data_lock_);
    SwapSchemaUnlocked(std::move(schema), version, &old_schema);
  }
}

void TabletMetadata::SwapSchemaUnlocked(SchemaPtr schema, uint32_t version, SchemaPtr* old_schema) {
  *old_schema = std::move(schema_);
  schema_ = std::move(schema);
  schema_version_ = version;
}


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc
File src/kudu/tserver/tserver_path_handlers.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tserver/tserver_path_handlers.cc@368
PS3, Line 368: *tmeta->schema()
Don't we need to make a schema_ptr here?



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 5
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Feb 2022 03:09:18 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#7).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/7
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 7
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 12:

(1 comment)

> Patch Set 11: Code-Review+1
> 
> (1 comment)

http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18255/11//COMMIT_MSG@23
PS11, Line 23: Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
             : 
             : 
             : 
             : 
             : 
> Remove these lines
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 12
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Mar 2022 11:08:30 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#12).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/12
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 12
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Reviewed-on: http://gerrit.cloudera.org:8080/18255
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 161 insertions(+), 149 deletions(-)

Approvals:
  Kudu Jenkins: Verified
  Andrew Wong: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 13
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#3).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
27 files changed, 137 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/3
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 3
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 3:

(2 comments)

Good work!

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc
File src/kudu/tablet/tablet.cc:

http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@548
PS3, Line 548: schema().get()
Why there are two styles, one is declear a new variable, like 'SchemaPtr schema_ptr = schema()', while another use 'schema().get()' directly?


http://gerrit.cloudera.org:8080/#/c/18255/3/src/kudu/tablet/tablet.cc@1616
PS3, Line 1616:   SchemaPtr schema(new Schema(new_schema));
How about use std::make_shared?



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 3
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Feb 2022 11:59:45 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 13:

Thanks for the contribution!


-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 13
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Mar 2022 00:17:02 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Hello Yingchun Lai, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18255

to look at the new patch set (#8).

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................

KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr

Change TabletMeta's variable Schema* to std::shared_ptr<Schema>
to reduce memory used when alter schema.

Because TabletMeta save old_schemas to reserve the elder schemas
when alter schema, maybe they have been used by scanners or
compaction jobs. As jira KUDU-3197 said, frequently alter schema will
lead to tserver's memory becomes very large, just like memory leak,
especially column's number is very large.

The jira issued by wangningito, and I continue his work, and
now use std::shared_ptr instead of scoped_refptr<Schema>, because
scoped_refptr<Schema> causes too many changes, just as:
https://gerrit.cloudera.org/c/18098/

Conflicts:
	src/kudu/master/sys_catalog.cc
	src/kudu/tablet/tablet.cc
	src/kudu/tablet/tablet_metadata.cc
	src/kudu/tserver/tablet_service.cc

Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/linked_list-test-util.h
M src/kudu/master/sys_catalog.cc
M src/kudu/tablet/all_types-scan-correctness-test.cc
M src/kudu/tablet/cfile_set.cc
M src/kudu/tablet/cfile_set.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/ops/alter_schema_op.cc
M src/kudu/tablet/ops/alter_schema_op.h
M src/kudu/tablet/ops/op.h
M src/kudu/tablet/ops/write_op.h
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet-schema-test.cc
M src/kudu/tablet/tablet-test-util.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_replica-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/scanners.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver_path_handlers.cc
30 files changed, 163 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/55/18255/8
-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 8
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>

[kudu-CR] KUDU-3197 [tserver] optimal Schema's memory used, using std::shared ptr

Posted by "Yuqi Du (Code Review)" <ge...@cloudera.org>.
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/18255 )

Change subject: KUDU-3197 [tserver] optimal Schema's memory used, using std::shared_ptr
......................................................................


Patch Set 9:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h
File src/kudu/tablet/tablet_metadata.h:

http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@379
PS8, Line 379: Sch
> nit: maybe remove the Set here, so it's just SwapSchemaUnlocked
Done


http://gerrit.cloudera.org:8080/#/c/18255/8/src/kudu/tablet/tablet_metadata.h@447
PS8, Line 447:   SchemaPtr schema_;
> nit: I'm not sure what this  means. There doesn't seem to be much of a diff
This is an old comment. I will remove it.



-- 
To view, visit http://gerrit.cloudera.org:8080/18255
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic284dde108c49130419d876c6698b40c195e9b35
Gerrit-Change-Number: 18255
Gerrit-PatchSet: 9
Gerrit-Owner: Yuqi Du <sh...@gmail.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <ac...@gmail.com>
Gerrit-Reviewer: Yuqi Du <sh...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Mar 2022 14:20:09 +0000
Gerrit-HasComments: Yes