You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/12/16 14:59:08 UTC
[GitHub] [ozone] bshashikant opened a new pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
bshashikant opened a new pull request #1717:
URL: https://github.com/apache/ozone/pull/1717
## What changes were proposed in this pull request?
1) Ensure single node ratis pipeline is not accounted for determining the pipeline limit
2) Made ozone.datanode.pipeline.limit to 0 to ensure no of pipelines created is to be determined by no raft log disks on a datanode
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4605
## How was this patch tested?
Modified UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #1717:
URL: https://github.com/apache/ozone/pull/1717#discussion_r552415835
##########
File path: hadoop-hdds/common/src/main/resources/ozone-default.xml
##########
@@ -777,9 +777,11 @@
</property>
<property>
<name>ozone.scm.datanode.pipeline.limit</name>
- <value>2</value>
+ <value>0</value>
<tag>OZONE, SCM, PIPELINE</tag>
<description>Max number of pipelines per datanode can be engaged in.
+ Setting the value to 0 means the pipeline limit per dn will be determined
+ by the o of metadata volumes reported per dn.
Review comment:
typo
```suggestion
by the number of metadata volumes reported per dn.
```
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java
##########
@@ -35,11 +35,7 @@
import org.apache.hadoop.hdds.protocol.DatanodeDetails;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.ratis.conf.RatisClientConfig;
-import org.apache.hadoop.hdds.scm.OzoneClientConfig;
-import org.apache.hadoop.hdds.scm.XceiverClientManager;
-import org.apache.hadoop.hdds.scm.XceiverClientRatis;
-import org.apache.hadoop.hdds.scm.XceiverClientReply;
-import org.apache.hadoop.hdds.scm.XceiverClientSpi;
+import org.apache.hadoop.hdds.scm.*;
Review comment:
Star import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1717:
URL: https://github.com/apache/ozone/pull/1717#issuecomment-756679252
The test failures seem to be resulting out of space issue in test environement.
`StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: java.io.IOException: No space left on device
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.wrapInStorageContainerException(ChunkUtils.java:331)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:125)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:110)
at org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy.writeChunk(FilePerBlockStrategy.java:120)
at org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerDispatcher.writeChunk(ChunkManagerDispatcher.java:70)
at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleWriteChunk(KeyValueHandler.java:722)
at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:199)
at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:163)
at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:309)
at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:171)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:170)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:398)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:408)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$2(ContainerStateMachine.java:445)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.FileDispatcherImpl.writev(FileDispatcherImpl.java:72)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:239)
at java.nio.channels.FileChannel.write(FileChannel.java:414)
at org.apache.hadoop.ozone.common.ChunkBufferImplWithByteBufferList.writeTo(ChunkBufferImplWithByteBufferList.java:186)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeDataToChannel(ChunkUtils.java:163)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.lambda$writeData$1(ChunkUtils.java:111)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:123)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant merged pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #1717:
URL: https://github.com/apache/ozone/pull/1717
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1717:
URL: https://github.com/apache/ozone/pull/1717#issuecomment-756679252
The test failures seem to be resulting out of space issue in test environement.
`StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: java.io.IOException: No space left on device
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.wrapInStorageContainerException(ChunkUtils.java:331)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:125)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:110)
at org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy.writeChunk(FilePerBlockStrategy.java:120)
at org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerDispatcher.writeChunk(ChunkManagerDispatcher.java:70)
at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleWriteChunk(KeyValueHandler.java:722)
at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:199)
at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:163)
at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:309)
at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:171)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:170)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:398)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:408)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$2(ContainerStateMachine.java:445)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.FileDispatcherImpl.writev(FileDispatcherImpl.java:72)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:239)
at java.nio.channels.FileChannel.write(FileChannel.java:414)
at org.apache.hadoop.ozone.common.ChunkBufferImplWithByteBufferList.writeTo(ChunkBufferImplWithByteBufferList.java:186)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeDataToChannel(ChunkUtils.java:163)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.lambda$writeData$1(ChunkUtils.java:111)
at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:123)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #1717:
URL: https://github.com/apache/ozone/pull/1717#issuecomment-756521522
> @bshashikant Thanks for working on the PR! The changes look good to me. Do we need to set the config `ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT` in tests? It should work by default? I have other minor comments inline.
@lokeshj1703 , thanks for looking into this. I have added the limit on some specific tests and these tests need only one pipeline to function properly. Enabling multiRaft by default for these break the assumptions made in the tests,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant merged pull request #1717: HDDS-4605. Enable Multi Raft by default in Ozone.
Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #1717:
URL: https://github.com/apache/ozone/pull/1717
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org