You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "whbing (via GitHub)" <gi...@apache.org> on 2023/03/30 15:26:31 UTC
[GitHub] [ozone] whbing opened a new pull request, #4503: HDDS-8331. Add rocksdb option template file and docs
whbing opened a new pull request, #4503:
URL: https://github.com/apache/ozone/pull/4503
## What changes were proposed in this pull request?
Add a template file and description to clarify how to customize the rocksdb options.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] github-actions[bot] commented on pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4503:
URL: https://github.com/apache/ozone/pull/4503#issuecomment-1610395371
Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #4503:
URL: https://github.com/apache/ozone/pull/4503#issuecomment-1504605416
Personally I think adding a template is fine. I was just trying to confirm there's no license issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] github-actions[bot] closed pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4503: HDDS-8331. Add rocksdb option template file and docs
URL: https://github.com/apache/ozone/pull/4503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on a diff in pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on code in PR #4503:
URL: https://github.com/apache/ozone/pull/4503#discussion_r1154024809
##########
hadoop-ozone/dist/src/shell/conf/db.ini.template:
##########
@@ -0,0 +1,25 @@
+# This is a RocksDB option file.
Review Comment:
> We should include a license header in this file to pass the RAT check.
> Is this file copied from RocksDB?
Copied from https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini, and added some description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #4503:
URL: https://github.com/apache/ozone/pull/4503#issuecomment-1494616645
cc @tanvipenumudy can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a diff in pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #4503:
URL: https://github.com/apache/ozone/pull/4503#discussion_r1153971658
##########
hadoop-ozone/dist/src/shell/conf/db.ini.template:
##########
@@ -0,0 +1,25 @@
+# This is a RocksDB option file.
Review Comment:
We should include a license header in this file to pass the RAT check.
Is this file copied from RocksDB?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a diff in pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #4503:
URL: https://github.com/apache/ozone/pull/4503#discussion_r1153968625
##########
hadoop-ozone/dist/src/shell/conf/db.ini.template:
##########
@@ -0,0 +1,25 @@
+# This is a RocksDB option file.
+#
+# For detailed file format spec, please refer to the example file
+# examples/rocksdb_option_file_example.ini in
+# https://github.com/facebook/rocksdb
+#
+# Below is the server name and corresponding RocksDB option file name:
+# OM om.db.ini
+# SCM scm.db.ini
+# DN dn-crl.db.ini
+#
+# Below is an example of a RocksDB options file:
+
+[DBOptions]
+ max_background_flushes=2
+ compaction_readahead_size=4194304
+ bytes_per_sync=1048576
+ max_background_compactions=4
+ max_background_jobs=48
+ create_missing_column_families=true
+ create_if_missing=true
+
+[CFOptions "default"]
+
+[TableOptions/BlockBasedTable "default"]
Review Comment:
Add a newline before EOF.
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBConfigFromFile.java:
##########
@@ -120,7 +120,7 @@ public static ManagedDBOptions readFromFile(String dbFileName,
Preconditions.checkNotNull(cfDescs);
Preconditions.checkArgument(cfDescs.size() > 0);
- //TODO: Add Documentation on how to support RocksDB Mem Env.
+ // Documentation on how to support RocksDB Mem Env: see HDDS-8331.
Review Comment:
Maybe direct link is better?
```suggestion
// Documentation on how to support RocksDB Mem Env:
// see hadoop-hdds/docs/content/recipe/RocksdbOptions.md
```
##########
hadoop-hdds/docs/content/recipe/RocksdbOptions.zh.md:
##########
@@ -0,0 +1,55 @@
+---
+title: 自定义Rocksdb参数
+linktitle: Rocksdb Options
+summary: OM/SCM/DN使用的自定义的Rocksdb参数
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+Ozone使用的rocksdb参数大多数为默认参数。本页介绍如何通过配置文件来使用自定义的rocksdb参数。
+
+## 使用Rocksdb配置文件
+
+将对应的配置文件`file`放在OZONE的配置目录`$OZONE_DIR/etc/hadoop`下，然后重启服务即可。以下列出了OM/SCM/DN默认加载的rocksdb配置文件名。
+
+服务 | 文件名
+----|---------
+OM | om.db.ini
+SCM | scm.db.ini
+DN | dn-crl.db.ini
+
+Rocksdb配置文件格式请参考：[rocksdb_option_file_example.ini](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini)
+
+## 验证配置文件生效
+
+查看rocksdb目录下的`OPTIONS-xxxxxx`相应配置项是否符合预期。例如，OM的配置文件`$OZONE_DIR/etc/hadoop/om.db.ini`如下：(修改`max_background_flushes=4`)
+
+```ini
+[DBOptions]
+ max_background_flushes=4
+ compaction_readahead_size=4194304
+ bytes_per_sync=1048576
+ max_background_compactions=4
+ max_background_jobs=48
+ create_missing_column_families=true
+ create_if_missing=true
+
+[CFOptions "default"]
+
+[TableOptions/BlockBasedTable "default"]
+```
+重启OM后，查看OM的rocksdb目录（`${DBDATA}/om.db`）下`OPTIONS-xxxxxx`内容，`max_background_flushes=4`表明配置生效。
Review Comment:
Please add a space between Chinese characters and English characters.
For example:
```
将对应的配置文件 `file` 放在OZONE的配置目录 `$OZONE_DIR/etc/hadoop` 下，然后重启服务即可。以下列出了 OM/SCM/DN 默认加载的 rocksdb 配置文件名。
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #4503:
URL: https://github.com/apache/ozone/pull/4503#issuecomment-1491409789
@kaijchen Thanks! add new commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a diff in pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #4503:
URL: https://github.com/apache/ozone/pull/4503#discussion_r1154092874
##########
hadoop-ozone/dist/src/shell/conf/db.ini.template:
##########
@@ -0,0 +1,25 @@
+# This is a RocksDB option file.
Review Comment:
Maybe we should keep the original author, and add this file in LICENSE.
Cc @adoroszlai, what do you think?
For example
https://github.com/apache/ozone/blob/57d66137fa54d566752f2787e853deecda9d194b/LICENSE.txt#L216-L217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #4503: HDDS-8331. Add rocksdb option template file and docs
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #4503:
URL: https://github.com/apache/ozone/pull/4503#issuecomment-1504553628
@kaijchen @kerneltime Adding a template file seems controversial. Just adding docs without template file maybe better.
I will remove the template file if possible. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org