You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "ShadowySpirits (via GitHub)" <gi...@apache.org> on 2023/01/26 02:22:43 UTC
[GitHub] [rocketmq] ShadowySpirits opened a new pull request, #5926: [ISSUE #5923] Add example tiered storage backend service provider
ShadowySpirits opened a new pull request, #5926:
URL: https://github.com/apache/rocketmq/pull/5926
close #5923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ShadowySpirits commented on a diff in pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "ShadowySpirits (via GitHub)" <gi...@apache.org>.
ShadowySpirits commented on code in PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926#discussion_r1092720499
##########
tieredstore/src/main/java/org/apache/rocketmq/tieredstore/provider/TieredStoreProvider.java:
##########
@@ -19,7 +19,7 @@
import java.nio.ByteBuffer;
import java.util.concurrent.CompletableFuture;
-public interface TieredStoreBackendProvider {
Review Comment:
The SPI mechanism is not applicable in this case. We want users to build their own provider by extending TieredFileSegment, which is an abstract class, not an interface. And tiered storage will create multiple file segment instances in multiple threads. It's more suitable to use the class name to initialize the file segment objects directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ShadowySpirits commented on pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "ShadowySpirits (via GitHub)" <gi...@apache.org>.
ShadowySpirits commented on PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926#issuecomment-1404649001
Please take a look. @aaron-ai @lollipopjin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lollipopjin merged pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "lollipopjin (via GitHub)" <gi...@apache.org>.
lollipopjin merged PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926#issuecomment-1404581037
# [Codecov](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5926](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (46a333c) into [develop](https://codecov.io/gh/apache/rocketmq/commit/a961a9b37b0e9e4c6e30d7cc6ab1a9c796634e6f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a961a9b) will **increase** coverage by `0.06%`.
> The diff coverage is `66.90%`.
```diff
@@ Coverage Diff @@
## develop #5926 +/- ##
=============================================
+ Coverage 43.36% 43.42% +0.06%
- Complexity 8810 8833 +23
=============================================
Files 1090 1091 +1
Lines 76407 76544 +137
Branches 9981 9989 +8
=============================================
+ Hits 33135 33241 +106
- Misses 39126 39159 +33
+ Partials 4146 4144 -2
```
| [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ocketmq/tieredstore/container/TieredIndexFile.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL2NvbnRhaW5lci9UaWVyZWRJbmRleEZpbGUuamF2YQ==) | `68.10% <ø> (-3.88%)` | :arrow_down: |
| [...ieredstore/metrics/TieredStoreMetricsConstant.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL21ldHJpY3MvVGllcmVkU3RvcmVNZXRyaWNzQ29uc3RhbnQuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...cketmq/tieredstore/provider/TieredFileSegment.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL3Byb3ZpZGVyL1RpZXJlZEZpbGVTZWdtZW50LmphdmE=) | `74.71% <ø> (ø)` | |
| [...che/rocketmq/tieredstore/util/TieredStoreUtil.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL3V0aWwvVGllcmVkU3RvcmVVdGlsLmphdmE=) | `67.21% <ø> (ø)` | |
| [...q/tieredstore/provider/posix/PosixFileSegment.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL3Byb3ZpZGVyL3Bvc2l4L1Bvc2l4RmlsZVNlZ21lbnQuamF2YQ==) | `60.17% <60.17%> (ø)` | |
| [...mq/tieredstore/metadata/TieredMetadataManager.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL21ldGFkYXRhL1RpZXJlZE1ldGFkYXRhTWFuYWdlci5qYXZh) | `85.39% <91.66%> (+2.75%)` | :arrow_up: |
| [...q/tieredstore/common/TieredMessageStoreConfig.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL2NvbW1vbi9UaWVyZWRNZXNzYWdlU3RvcmVDb25maWcuamF2YQ==) | `70.45% <100.00%> (+0.92%)` | :arrow_up: |
| [...store/metadata/TieredMetadataSerializeWrapper.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-dGllcmVkc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3RpZXJlZHN0b3JlL21ldGFkYXRhL1RpZXJlZE1ldGFkYXRhU2VyaWFsaXplV3JhcHBlci5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...q/namesrv/routeinfo/BrokerHousekeepingService.java](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bmFtZXNydi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcm9ja2V0bXEvbmFtZXNydi9yb3V0ZWluZm8vQnJva2VySG91c2VrZWVwaW5nU2VydmljZS5qYXZh) | `70.00% <0.00%> (-10.00%)` | :arrow_down: |
| ... and [20 more](https://codecov.io/gh/apache/rocketmq/pull/5926?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lollipopjin commented on a diff in pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "lollipopjin (via GitHub)" <gi...@apache.org>.
lollipopjin commented on code in PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926#discussion_r1091812269
##########
tieredstore/src/main/java/org/apache/rocketmq/tieredstore/provider/posix/PosixFileSegment.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.tieredstore.provider.posix;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.io.ByteStreams;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.tieredstore.common.TieredMessageStoreConfig;
+import org.apache.rocketmq.tieredstore.common.TieredStoreExecutor;
+import org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsManager;
+import org.apache.rocketmq.tieredstore.provider.TieredFileSegment;
+import org.apache.rocketmq.tieredstore.util.TieredStoreUtil;
+
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_FILE_TYPE;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_OPERATION;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_SUCCESS;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_TOPIC;
+
+public class PosixFileSegment extends TieredFileSegment {
+ private static final Logger logger = LoggerFactory.getLogger(TieredStoreUtil.TIERED_STORE_LOGGER_NAME);
+
+ private static final String OPERATION_POSIX_READ = "read";
+ private static final String OPERATION_POSIX_WRITE = "write";
+
+ private final String basePath;
+ private final String filepath;
+
+ private volatile File file;
+ private volatile FileChannel readFileChannel;
+ private volatile FileChannel writeFileChannel;
+
+ public PosixFileSegment(FileSegmentType fileType, MessageQueue messageQueue,
+ long baseOffset, TieredMessageStoreConfig storeConfig) {
+ super(fileType, messageQueue, baseOffset, storeConfig);
+
+ String basePath = storeConfig.getTieredStoreFilepath();
+ if (StringUtils.isBlank(basePath) || basePath.endsWith(File.separator)) {
+ this.basePath = basePath;
+ } else {
+ this.basePath = basePath + File.separator;
+ }
+ this.filepath = this.basePath
Review Comment:
Take filePath length into condition？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ShadowySpirits commented on a diff in pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "ShadowySpirits (via GitHub)" <gi...@apache.org>.
ShadowySpirits commented on code in PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926#discussion_r1092723138
##########
tieredstore/src/main/java/org/apache/rocketmq/tieredstore/provider/posix/PosixFileSegment.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.tieredstore.provider.posix;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.io.ByteStreams;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.tieredstore.common.TieredMessageStoreConfig;
+import org.apache.rocketmq.tieredstore.common.TieredStoreExecutor;
+import org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsManager;
+import org.apache.rocketmq.tieredstore.provider.TieredFileSegment;
+import org.apache.rocketmq.tieredstore.util.TieredStoreUtil;
+
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_FILE_TYPE;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_OPERATION;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_SUCCESS;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_TOPIC;
+
+public class PosixFileSegment extends TieredFileSegment {
+ private static final Logger logger = LoggerFactory.getLogger(TieredStoreUtil.TIERED_STORE_LOGGER_NAME);
+
+ private static final String OPERATION_POSIX_READ = "read";
+ private static final String OPERATION_POSIX_WRITE = "write";
+
+ private final String basePath;
+ private final String filepath;
+
+ private volatile File file;
+ private volatile FileChannel readFileChannel;
+ private volatile FileChannel writeFileChannel;
+
+ public PosixFileSegment(FileSegmentType fileType, MessageQueue messageQueue,
+ long baseOffset, TieredMessageStoreConfig storeConfig) {
+ super(fileType, messageQueue, baseOffset, storeConfig);
+
+ String basePath = storeConfig.getTieredStoreFilepath();
+ if (StringUtils.isBlank(basePath) || basePath.endsWith(File.separator)) {
+ this.basePath = basePath;
+ } else {
+ this.basePath = basePath + File.separator;
+ }
+ this.filepath = this.basePath
Review Comment:
The maximum length for a file name is 255 bytes on Linux. And it is 4096 bytes for the file path. This length is enough, no need to worry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] aaron-ai commented on a diff in pull request #5926: [ISSUE #5923] Add example tiered storage backend service provider
Posted by "aaron-ai (via GitHub)" <gi...@apache.org>.
aaron-ai commented on code in PR #5926:
URL: https://github.com/apache/rocketmq/pull/5926#discussion_r1090139777
##########
tieredstore/src/main/java/org/apache/rocketmq/tieredstore/provider/TieredStoreProvider.java:
##########
@@ -19,7 +19,7 @@
import java.nio.ByteBuffer;
import java.util.concurrent.CompletableFuture;
-public interface TieredStoreBackendProvider {
Review Comment:
Could we use Java SPI mechanism to get the implementation?
##########
tieredstore/src/main/java/org/apache/rocketmq/tieredstore/provider/posix/PosixFileSegment.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.tieredstore.provider.posix;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.io.ByteStreams;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.rocketmq.common.message.MessageQueue;
+import org.apache.rocketmq.logging.org.slf4j.Logger;
+import org.apache.rocketmq.logging.org.slf4j.LoggerFactory;
+import org.apache.rocketmq.tieredstore.common.TieredMessageStoreConfig;
+import org.apache.rocketmq.tieredstore.common.TieredStoreExecutor;
+import org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsManager;
+import org.apache.rocketmq.tieredstore.provider.TieredFileSegment;
+import org.apache.rocketmq.tieredstore.util.TieredStoreUtil;
+
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_FILE_TYPE;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_OPERATION;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_SUCCESS;
+import static org.apache.rocketmq.tieredstore.metrics.TieredStoreMetricsConstant.LABEL_TOPIC;
+
+public class PosixFileSegment extends TieredFileSegment {
Review Comment:
According to readme, It seems that `PosixFileSegment` is just an example, there should be more comments to illustrate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org