You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "KeDeng (Code Review)" <ge...@cloudera.org> on 2023/08/03 09:21:24 UTC
[kudu-CR] [multi-tenancy] Support for tenants to have independent directories
KeDeng has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20309
Change subject: [multi-tenancy] Support for tenants to have independent directories
......................................................................
[multi-tenancy] Support for tenants to have independent directories
This patch belongs to the multi-tenancy feature. In this patch,
I have set up different directories for different tenants to
facilitate the development and maintenance of subsequent features.
When the multi-tenancy feature is disabled, the data directory tree
will be as follows:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ ├── block_manager_instance
│ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.data
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.metadata
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.data
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.metadata
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.data
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.metadata
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.data
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.metadata
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ └── fff0b5cabbe8444db30a7719490d0214.metadata
│ └── instance
└── tserver_wal
├── consensus-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
└── wals
├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
├── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
When the multi-tenancy feature is enabled, the data directory tree
will change to the following form:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ └── block_manager_instance
│ ├── instance
│ └── tenants
│ ├── 00000000000000000000000000000001
│ │ ├── data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ │ └── block_manager_instance
│ │ └── instance
│ └── 00000000000000000000000000000002
│ ├── data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.metadata
│ │ └── block_manager_instance
│ └── instance
└── tserver_wal
├── consensus-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── instance
├── tablet-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── wals
│ └── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
└── tenants
├── 00000000000000000000000000000001
│ ├── consensus-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ ├── instance
│ ├── tablet-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ └── wals
│ └── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 00000000000000000000000000000002
├── consensus-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
└── wals
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
As it involves tenant directory management, I have added some logic
while ensuring that existing logic is not affected, and have also
added some tests to ensure that this logic is effective.
Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
---
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/tablet/delta_compaction.cc
5 files changed, 781 insertions(+), 265 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/09/20309/1
--
To view, visit http://gerrit.cloudera.org:8080/20309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
Gerrit-Change-Number: 20309
Gerrit-PatchSet: 1
Gerrit-Owner: KeDeng <kd...@gmail.com>
[kudu-CR] KUDU-3413 [multi-tenancy] Support for tenant storage path isolation
Posted by "Ádám Bakai (Code Review)" <ge...@cloudera.org>.
Ádám Bakai has posted comments on this change. ( http://gerrit.cloudera.org:8080/20309 )
Change subject: KUDU-3413 [multi-tenancy] Support for tenant storage path isolation
......................................................................
Patch Set 5: Code-Review-1
(1 comment)
I tried running a cluster in a clear environment, and I got an error:
I20240103 12:56:00.630818 2130065 server_base.cc:767] This appears to be a new deployment of Kudu; creating new FS layout
F20240103 12:56:00.631273 2130065 map-util.h:98] Check failed: it != collection.end() Map key not found: 876
*** Check failure stack trace: ***
Signal: SIGABRT (Aborted)
the relevant command line flags are: --enable_multi_tenancy=true --encrypt_data_at_rest=true --unlock_experimental_flags --test-tenant-id=876 --test-tenant-key=test --test-tenant-name=test --test-tenant-key-iv=test --test-tenant-key-version=1"
I understand that the tenant directory doesn't exist yet, but I think it should create it instead of throwing an error.
http://gerrit.cloudera.org:8080/#/c/20309/5/src/kudu/fs/fs_manager.h
File src/kudu/fs/fs_manager.h:
http://gerrit.cloudera.org:8080/#/c/20309/5/src/kudu/fs/fs_manager.h@283
PS5, Line 283: std::string GetWalsRootDir(
This is a general comment rather than a comment about this special line. I think setting tenant at multiple lines is an antipattern, and if tenant_id will be used by then every place where these functions are called, tenant_id should be needed to pass on.
I have two possible ideas:
1. create a subclass from FSManager that handles and encapsulates tenant data, and that object will be passed down to the objects that use them.
2. If I understand correctly there can be a tablet_id->tenant_id mapping, and tablet_id is already propagated to everywhere, and getting the tenant_id from some kind of map instead of as an argument would make the code less complex.
(I checked if GetWalsRootDir can get a tablet_id instead of tenant_id, and it looks like it is possible with one exception:
int64_t wal_dir_space = CalculateAvailableSpace(options, fs_manager.GetWalsRootDir(), FLAGS_fs_wal_dir_reserved_bytes, &space_info_waldir);
which raises the question: when any kind of available space calculation happens, should we calculate all of the tenants together? Should they be calculated separately? My intuition that they should be calculated together. )
--
To view, visit http://gerrit.cloudera.org:8080/20309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
Gerrit-Change-Number: 20309
Gerrit-PatchSet: 5
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 12:44:46 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3413 [multi-tenancy] Support for tenant storage path isolation
Posted by "KeDeng (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Yingchun Lai, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20309
to look at the new patch set (#5).
Change subject: KUDU-3413 [multi-tenancy] Support for tenant storage path isolation
......................................................................
KUDU-3413 [multi-tenancy] Support for tenant storage path isolation
This patch belongs to the multi-tenancy feature. In this patch,
I have set up different directories for different tenants to
facilitate the development and maintenance of subsequent features.
When the multi-tenancy feature is disabled, the data directory tree
will be as follows:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ ├── block_manager_instance
│ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.data
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.metadata
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.data
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.metadata
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.data
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.metadata
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.data
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.metadata
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ └── fff0b5cabbe8444db30a7719490d0214.metadata
│ └── instance
└── tserver_wal
├── consensus-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
└── wals
├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
├── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
When the multi-tenancy feature is enabled, the data directory tree
will change to the following form:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ └── block_manager_instance
│ ├── instance
│ └── tenants
│ ├── 00000000000000000000000000000001
│ │ ├── data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ │ └── block_manager_instance
│ │ └── instance
│ └── 00000000000000000000000000000002
│ ├── data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.metadata
│ │ └── block_manager_instance
│ └── instance
└── tserver_wal
├── consensus-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── instance
├── tablet-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── wals
│ └── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
└── tenants
├── 00000000000000000000000000000001
│ ├── consensus-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ ├── instance
│ ├── tablet-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ └── wals
│ └── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 00000000000000000000000000000002
├── consensus-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
└── wals
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
As it involves tenant directory management, I have added some logic
while ensuring that existing logic is not affected, and have also
added some tests to ensure that this logic is effective.
Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
---
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/tablet/delta_compaction.cc
M src/kudu/tools/tool_action_fs.cc
6 files changed, 792 insertions(+), 275 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/09/20309/5
--
To view, visit http://gerrit.cloudera.org:8080/20309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
Gerrit-Change-Number: 20309
Gerrit-PatchSet: 5
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
[kudu-CR] [multi-tenancy] Support for tenants to have independent directories
Posted by "KeDeng (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20309
to look at the new patch set (#3).
Change subject: [multi-tenancy] Support for tenants to have independent directories
......................................................................
[multi-tenancy] Support for tenants to have independent directories
This patch belongs to the multi-tenancy feature. In this patch,
I have set up different directories for different tenants to
facilitate the development and maintenance of subsequent features.
When the multi-tenancy feature is disabled, the data directory tree
will be as follows:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ ├── block_manager_instance
│ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.data
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.metadata
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.data
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.metadata
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.data
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.metadata
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.data
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.metadata
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ └── fff0b5cabbe8444db30a7719490d0214.metadata
│ └── instance
└── tserver_wal
├── consensus-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
└── wals
├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
├── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
When the multi-tenancy feature is enabled, the data directory tree
will change to the following form:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ └── block_manager_instance
│ ├── instance
│ └── tenants
│ ├── 00000000000000000000000000000001
│ │ ├── data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ │ └── block_manager_instance
│ │ └── instance
│ └── 00000000000000000000000000000002
│ ├── data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.metadata
│ │ └── block_manager_instance
│ └── instance
└── tserver_wal
├── consensus-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── instance
├── tablet-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── wals
│ └── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
└── tenants
├── 00000000000000000000000000000001
│ ├── consensus-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ ├── instance
│ ├── tablet-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ └── wals
│ └── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 00000000000000000000000000000002
├── consensus-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
└── wals
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
As it involves tenant directory management, I have added some logic
while ensuring that existing logic is not affected, and have also
added some tests to ensure that this logic is effective.
Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
---
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/tablet/delta_compaction.cc
5 files changed, 781 insertions(+), 265 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/09/20309/3
--
To view, visit http://gerrit.cloudera.org:8080/20309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
Gerrit-Change-Number: 20309
Gerrit-PatchSet: 3
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] [multi-tenancy] Support for tenants to have independent directories
Posted by "KeDeng (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20309
to look at the new patch set (#4).
Change subject: [multi-tenancy] Support for tenants to have independent directories
......................................................................
[multi-tenancy] Support for tenants to have independent directories
This patch belongs to the multi-tenancy feature. In this patch,
I have set up different directories for different tenants to
facilitate the development and maintenance of subsequent features.
When the multi-tenancy feature is disabled, the data directory tree
will be as follows:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ ├── block_manager_instance
│ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.data
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.metadata
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.data
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.metadata
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.data
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.metadata
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.data
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.metadata
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ └── fff0b5cabbe8444db30a7719490d0214.metadata
│ └── instance
└── tserver_wal
├── consensus-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
└── wals
├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
├── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
When the multi-tenancy feature is enabled, the data directory tree
will change to the following form:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ └── block_manager_instance
│ ├── instance
│ └── tenants
│ ├── 00000000000000000000000000000001
│ │ ├── data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ │ └── block_manager_instance
│ │ └── instance
│ └── 00000000000000000000000000000002
│ ├── data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.metadata
│ │ └── block_manager_instance
│ └── instance
└── tserver_wal
├── consensus-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── instance
├── tablet-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── wals
│ └── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
└── tenants
├── 00000000000000000000000000000001
│ ├── consensus-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ ├── instance
│ ├── tablet-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ └── wals
│ └── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 00000000000000000000000000000002
├── consensus-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
└── wals
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
As it involves tenant directory management, I have added some logic
while ensuring that existing logic is not affected, and have also
added some tests to ensure that this logic is effective.
Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
---
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/tablet/delta_compaction.cc
5 files changed, 781 insertions(+), 265 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/09/20309/4
--
To view, visit http://gerrit.cloudera.org:8080/20309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
Gerrit-Change-Number: 20309
Gerrit-PatchSet: 4
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] [multi-tenancy] Support for tenants to have independent directories
Posted by "KeDeng (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20309
to look at the new patch set (#2).
Change subject: [multi-tenancy] Support for tenants to have independent directories
......................................................................
[multi-tenancy] Support for tenants to have independent directories
This patch belongs to the multi-tenancy feature. In this patch,
I have set up different directories for different tenants to
facilitate the development and maintenance of subsequent features.
When the multi-tenancy feature is disabled, the data directory tree
will be as follows:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ ├── block_manager_instance
│ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.data
│ │ ├── d1aa8feaccd9448a846ff9bcbeea0143.metadata
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.data
│ │ ├── d39d42f6c25947f5a062a4e4003a33a3.metadata
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.data
│ │ ├── d549fa825f6641c898f2b5bcf92e43bb.metadata
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.data
│ │ ├── ed7e1ff546b9435c9cd793de0ef589cf.metadata
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ └── fff0b5cabbe8444db30a7719490d0214.metadata
│ └── instance
└── tserver_wal
├── consensus-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ ├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── 276b446676f14f589c8e8434876e3ba1
│ ├── 45b3416b328741ceb64c73b4ea43aab0
└── wals
├── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
├── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
When the multi-tenancy feature is enabled, the data directory tree
will change to the following form:
`
/rnddata00/kudu/
├── tserver_data
│ ├── data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.data
│ │ ├── b83fe60fc9f541d9a09398b77d2a8fdb.metadata
│ │ └── block_manager_instance
│ ├── instance
│ └── tenants
│ ├── 00000000000000000000000000000001
│ │ ├── data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.data
│ │ │ ├── c3aad78253af427dad33641e6e092ec8.metadata
│ │ │ └── block_manager_instance
│ │ └── instance
│ └── 00000000000000000000000000000002
│ ├── data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.data
│ │ ├── fff0b5cabbe8444db30a7719490d0214.metadata
│ │ └── block_manager_instance
│ └── instance
└── tserver_wal
├── consensus-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── instance
├── tablet-meta
│ └── 0e904e919ba94dcf9f64f8ccca155c71
├── wals
│ └── 0e904e919ba94dcf9f64f8ccca155c71
│ ├── index.000000000
│ └── wal-000000001
└── tenants
├── 00000000000000000000000000000001
│ ├── consensus-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ ├── instance
│ ├── tablet-meta
│ │ └── 276b446676f14f589c8e8434876e3ba1
│ └── wals
│ └── 276b446676f14f589c8e8434876e3ba1
│ ├── index.000000000
│ └── wal-000000001
└── 00000000000000000000000000000002
├── consensus-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
├── instance
├── tablet-meta
│ └── 45b3416b328741ceb64c73b4ea43aab0
└── wals
└── 45b3416b328741ceb64c73b4ea43aab0
├── index.000000000
└── wal-000000001
`
As it involves tenant directory management, I have added some logic
while ensuring that existing logic is not affected, and have also
added some tests to ensure that this logic is effective.
Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
---
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/tablet/delta_compaction.cc
5 files changed, 781 insertions(+), 265 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/09/20309/2
--
To view, visit http://gerrit.cloudera.org:8080/20309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb3f3d47e00f93db6f1ce66596de6449fb1dbb73
Gerrit-Change-Number: 20309
Gerrit-PatchSet: 2
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)