You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ratis.apache.org by GitBox <gi...@apache.org> on 2020/12/10 11:12:56 UTC
[GitHub] [incubator-ratis] runzhiwang opened a new pull request #347: RATIS-1229. FileStore client generate files in parallel
runzhiwang opened a new pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
## What changes were proposed in this pull request?
1. FileStore client generate files by multi-thread, in multi-disk
2. config write, read, commit, delete thread number in FileStoreStateMachine
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-1229
## How was this patch tested?
test it manually by command
client:
`${BIN}/client.sh filestore datastream --size 128000000 --numFiles 1000 --bufferSize 4000000 --syncSize 16000000 --type DirectByteBuffer --peers ${PEERS} --storage /data/ratis/n2 --storage /data1/ratis/n2 --storage /data2/ratis/n2 --storage /data3/ratis/n2 --storage /data4/ratis/n2 --storage /data5/ratis/n2 --storage /data6/ratis/n2 --storage /data7/ratis/n2 --storage /data8/ratis/n2 --storage /data9/ratis/n2 --storage /data10/ratis/n2 --storage /data11/ratis/n2`
server:
`nohup ${BIN}/server.sh filestore server --id n2 --storage /data/ratis/n2 --storage /data1/ratis/n2 --storage /data2/ratis/n2 --storage /data3/ratis/n2 --storage /data4/ratis/n2 --storage /data5/ratis/n2 --storage /data6/ratis/n2 --storage /data7/ratis/n2 --storage /data8/ratis/n2 --storage /data9/ratis/n2 --storage /data10/ratis/n2 --storage /data11/ratis/n2 --peers ${PEERS} --writeThreadNum 100 --readThreadNum 100 --commitThreadNum 20 --deleteThreadNum 20 >> n2.log 2>&1 &`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang closed pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang closed pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang closed pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] szetszwo commented on a change in pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347#discussion_r540179397
##########
File path: ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Client.java
##########
@@ -153,6 +199,7 @@ protected void writeFile(String path, int fileSize, int bufferSize, int random)
raf.close();
}
}
Review comment:
We may use ThreadLocalRandom.current().nextBytes(buffer) to fill the buffer.
```
final byte[] buffer = new byte[Math.toIntExact(bufferSize)];
try(RandomAccessFile raf = new RandomAccessFile(path, "rw")) {
while (offset < fileSize) {
final long remaining = fileSize - offset;
final long chunkSize = Math.min(remaining, bufferSize);
ThreadLocalRandom.current().nextBytes(buffer);
raf.write(buffer, 0, Math.toIntExact(chunkSize));
offset += chunkSize;
}
}
```
##########
File path: ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Client.java
##########
@@ -121,27 +133,61 @@ protected void stop(RaftClient client) throws IOException {
System.exit(0);
}
- protected List<String> generateFiles() throws IOException {
+ public String getPath(String fileName) {
+ int hash = fileName.hashCode() % storageDir.size();
+ return new File(storageDir.get(Math.abs(hash)), fileName).getAbsolutePath();
+ }
+
+ protected void dropCache() throws InterruptedException, IOException {
+ String[] cmds = {"/bin/sh","-c","echo 3 > /proc/sys/vm/drop_caches"};
+ Process pro = Runtime.getRuntime().exec(cmds);
+ pro.waitFor();
Review comment:
Let's try-catch it since the command is platform dependent.
```
try {
Process pro = Runtime.getRuntime().exec(cmds);
pro.waitFor();
} catch (Throwable t) {
LOG.warn("Failed to run command " + Arrays.toString(cmds), t);
}
```
##########
File path: ratis-examples/src/main/java/org/apache/ratis/examples/filestore/cli/Server.java
##########
@@ -62,6 +59,18 @@
required = true)
private List<File> storageDir = new ArrayList<>();
+ @Parameter(names = {"--writeThreadNum"}, description = "Number of write thread")
+ private int writeThreadNum = 10;
+
+ @Parameter(names = {"--readThreadNum"}, description = "Number of read thread")
+ private int readThreadNum = 10;
Review comment:
Let's set the default values of writeThreadNum and readThreadNum to 20.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang closed pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang closed pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang merged pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang merged pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-ratis] runzhiwang closed pull request #347: RATIS-1229. FileStore client generate files in parallel
Posted by GitBox <gi...@apache.org>.
runzhiwang closed pull request #347:
URL: https://github.com/apache/incubator-ratis/pull/347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org