You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/01/27 07:55:28 UTC
[GitHub] [ozone] guohao-rosicky opened a new pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
guohao-rosicky opened a new pull request #3023:
URL: https://github.com/apache/ozone/pull/3023
## What changes were proposed in this pull request?
Data Channel abstraction on datanode
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6229
## How was this patch tested?
see https://github.com/apache/ozone/pull/2860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc merged pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
captainzmc merged pull request #3023:
URL: https://github.com/apache/ozone/pull/3023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#issuecomment-1034589319
I have removed interface, and I will modify the implementation of putBlock in #2860 later.
@szetszwo Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#issuecomment-1040420397
By retriggering CI, I found that the test case reporting the error is irrelevant to the current change. The CI of HDDS-4454 branch is not stable at present, we can do rebase again after the master stable.
We can merge this PR and then continue the optimization of small file and put block. Thanks @guohao-rosicky for the contribution and thanks @szetszwo for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r794159843
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
@szetszwo
Like https://github.com/guohao-rosicky/ozone/commit/9ef1867b73bee4bbdb2bec8ad24a4e20032ff8c2#diff-d7517af5785df16a7d6507c34f75ddc93969875fa24250033aeba5886bb831feR127
I need to get the BCSID at the putBlock stage.
And inhttps://issues.apache.org/jira/browse/HDDS-6137
We will add the putBlock into stream and need to get the BCSID through the link method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#issuecomment-1040420397
By retriggering CI, I found that the test case reporting the error is irrelevant to the current change. The CI of HDDS-4454 branch is not stable at present, we can do rebase again after the master stable.
We can merge this PR and then continue the optimization of small file and put block. Thanks @guohao-rosicky for the contribution and thanks @szetszwo for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc merged pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
captainzmc merged pull request #3023:
URL: https://github.com/apache/ozone/pull/3023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r794379321
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
> I need to get the BCSID at the putBlock stage.
In ContainerStateMachine.link(..), build a DispatcherContext with the BCSID and then pass the DispatcherContext to submitTask(..), which will eventually call blockManager.putBlock(..).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r800367867
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
@szetszwo
I add ​putBlock into the channel, please check the logic of smallFileChannel, blockData is temporarily stored in the channel, so I need to return blockData in the channel, Is this bad for channel abstraction.
I added a comment in #2860, please see:
https://github.com/apache/ozone/pull/2860/files#r800366861
and
https://github.com/apache/ozone/pull/2860/files#r800366033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r794159843
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
@szetszwo
Like https://github.com/apache/ozone/pull/2860/files#diff-731ef3fb5d0f1e895fb7e91ee065e0a6177643164f9c6287134e3529cdf98a71R161
I need to get the BCSID at the putBlock stage.
And in https://issues.apache.org/jira/browse/HDDS-6137
We will add the putBlock into stream and need to get the BCSID through the link method.
See https://github.com/guohao-rosicky/ozone/commit/9ef1867b73bee4bbdb2bec8ad24a4e20032ff8c2#diff-d7517af5785df16a7d6507c34f75ddc93969875fa24250033aeba5886bb831feR127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r794159843
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
@szetszwo
https://issues.apache.org/jira/browse/HDDS-6137
We will add the putBlock into stream and need to get the BCSID through the link method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#issuecomment-1040468405
@captainzmc , thanks for merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r794162514
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/StreamDataChannelBase.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.keyvalue.impl;
+
+import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
+import org.apache.hadoop.ozone.container.common.helpers.ContainerMetrics;
+import org.apache.hadoop.ozone.container.common.impl.ContainerData;
+import org.apache.hadoop.ozone.container.common.transport.server.ratis.StreamDataChannel;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
+/**
+ * For write state machine data.
+ */
+abstract class StreamDataChannelBase implements StreamDataChannel {
+ private final RandomAccessFile randomAccessFile;
+
+ private final File file;
+
+ private final ContainerData containerData;
+ private final ContainerMetrics metrics;
+
+ StreamDataChannelBase(File file, ContainerData containerData,
+ ContainerMetrics metrics)
+ throws StorageContainerException {
+ try {
+ this.file = file;
+ this.randomAccessFile = new RandomAccessFile(file, "rw");
+ } catch (FileNotFoundException e) {
+ throw new StorageContainerException("BlockFile not exists with " +
+ "container Id " + containerData.getContainerID() +
+ " file " + file.getAbsolutePath(),
+ ContainerProtos.Result.IO_EXCEPTION);
+ }
+ this.containerData = containerData;
+ this.metrics = metrics;
+ }
+
+ protected File getFile() {
+ return file;
+ }
Review comment:
yes, I'm going to delete it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r793382087
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
Checked the #2860 . This link(..) method is for
```
blockManager.putBlock(kvContainer, blockData);
```
We should call ContainerStateMachine.submitTask(..) instead of blockManager.putBlock(..). Otherwise, it will skip all the steps in-between.
So, we should remove this method and also this the interface since the interface becomes empty.
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/StreamDataChannelBase.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.keyvalue.impl;
+
+import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
+import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
+import org.apache.hadoop.ozone.container.common.helpers.ContainerMetrics;
+import org.apache.hadoop.ozone.container.common.impl.ContainerData;
+import org.apache.hadoop.ozone.container.common.transport.server.ratis.StreamDataChannel;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
+/**
+ * For write state machine data.
+ */
+abstract class StreamDataChannelBase implements StreamDataChannel {
+ private final RandomAccessFile randomAccessFile;
+
+ private final File file;
+
+ private final ContainerData containerData;
+ private final ContainerMetrics metrics;
+
+ StreamDataChannelBase(File file, ContainerData containerData,
+ ContainerMetrics metrics)
+ throws StorageContainerException {
+ try {
+ this.file = file;
+ this.randomAccessFile = new RandomAccessFile(file, "rw");
+ } catch (FileNotFoundException e) {
+ throw new StorageContainerException("BlockFile not exists with " +
+ "container Id " + containerData.getContainerID() +
+ " file " + file.getAbsolutePath(),
+ ContainerProtos.Result.IO_EXCEPTION);
+ }
+ this.containerData = containerData;
+ this.metrics = metrics;
+ }
+
+ protected File getFile() {
+ return file;
+ }
Review comment:
getFile() is unused. Let's add it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#discussion_r800367867
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/StreamDataChannel.java
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.common.transport.server.ratis;
+
+import org.apache.ratis.proto.RaftProtos;
+import org.apache.ratis.statemachine.StateMachine;
+
+import java.io.IOException;
+
+/**
+ * For write state machine data.
+ */
+public interface StreamDataChannel extends StateMachine.DataChannel {
+
+ default void link(RaftProtos.LogEntryProto entry) throws IOException {
+ }
Review comment:
@szetszwo
I add ​putBlock into the channel, please check the logic of smallFileChannel, blockData is temporarily stored in the channel, so I need to return blockData in the channel, Is this bad for channel abstraction.
I added a comment in #2860, please see:
https://github.com/apache/ozone/pull/2860/files#r800366861
and
https://github.com/apache/ozone/pull/2860/files#r800366033
This helps us to discuss the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3023: HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #3023:
URL: https://github.com/apache/ozone/pull/3023#issuecomment-1040468405
@captainzmc , thanks for merging this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org