You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by GitBox <gi...@apache.org> on 2022/02/15 05:37:30 UTC
[GitHub] [ratis] guohao-rosicky opened a new pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
guohao-rosicky opened a new pull request #601:
URL: https://github.com/apache/ratis/pull/601
## What changes were proposed in this pull request?
Optional DataStreamManagement#startTransaction configuration
## What is the link to the Apache JIRA
see:
https://issues.apache.org/jira/browse/RATIS-1524
https://issues.apache.org/jira/browse/RATIS-1513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042585613
> > https://issues.apache.org/jira/browse/RATIS-1513
>
> @guohao-rosicky, @captainzmc, I really hope that we could fix "TimeoutIOException: Timeout 3000ms". How about we fix it first?
We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.
We have a test report showing that DataStreamManagement#startTransaction caused "TimeoutIOException: Timeout 3000ms".
@szetszwo I was hoping you could help us come up with a solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042521209
> > ... could we get a BCSID in another way than raft log?
>
> We need a BCSID and also the ability to commit transactions. I guess there are no easy ways. Otherwise, we can use it to replace the Raft Consensus Algorithm in general.
```
package org.apache.ratis.io;
public enum StandardWriteOption implements WriteOption {
/** Sync the data to the underlying storage. */
SYNC,
/** Close the data to the underlying storage. */
CLOSE,
/** Returns a unique ID **/
UNIQUE_ID,
}
```
@szetszwo
Can we add a new WriteOption? When the primary node receives this request, it generates a unique ID of type long, synchronizes it to the other nodes, and returns it to the client
In other words, generating an ID on the Primary node and passing it to the other nodes as a stream can improve throughput without raft requests internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042521209
> > ... could we get a BCSID in another way than raft log?
>
> We need a BCSID and also the ability to commit transactions. I guess there are no easy ways. Otherwise, we can use it to replace the Raft Consensus Algorithm in general.
```
package org.apache.ratis.io;
public enum StandardWriteOption implements WriteOption {
/** Sync the data to the underlying storage. */
SYNC,
/** Close the data to the underlying storage. */
CLOSE,
/** Returns a unique ID **/
UNIQUE_ID,
}
```
@szetszwo
Can we add a new WriteOption? When the primary node receives this request, it generates a unique ID of type long, synchronizes it to the other nodes, and returns it to the client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1039969825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042585613
> > https://issues.apache.org/jira/browse/RATIS-1513
>
> @guohao-rosicky, @captainzmc, I really hope that we could fix "TimeoutIOException: Timeout 3000ms". How about we fix it first?
We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.
We have a test report showing that DataStreamManagement#startTransaction caused "TimeoutIOException: Timeout 3000ms".
@szetszwo I was hoping you could help us come up with a solution.
The solution I can think of so far is to skip DataStreamManagement#startTransaction that triggers the timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1040022460
@szetszwo I want to discuss a question about ozone BCSID.
Does BCSID still make sense when you use stream to transfer data and put A Block into the stream process? Because raft transport is deprecated, playback of raft log is eliminated.
I can open a JIRA in Ozone and discuss this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042585613
> > https://issues.apache.org/jira/browse/RATIS-1513
>
> @guohao-rosicky, @captainzmc, I really hope that we could fix "TimeoutIOException: Timeout 3000ms". How about we fix it first?
We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.
We have a test report showing that DataStreamManagement#startTransaction caused "TimeoutIOException: Timeout 3000ms".
I was hoping you could help us come up with a solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042543901
I can open a JIRA in Ratis and discuss this @szetszwo @captainzmc
https://issues.apache.org/jira/browse/RATIS-1513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042543901
I can open a JIRA in Ratis and discuss this
https://issues.apache.org/jira/browse/RATIS-1513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042656987
@szetszwo Can you take a look at our test report and consider further optimizing the performance of ratis stream over ozone based on the problems identified in the test report.
Discuss how to optimize the scheme.
@captainzmc and I can participate in the development of optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1041143334
@guohao-rosicky , without BCSID and the Ratis log, the current Ozone design won't work for recovery. Why we need Ratis (or Raft) in Ozone? It is because Ratis provides a consistent view of the data among the servers. Without Ratis, the data in the servers may diverge. How could you tell which one to trust? And how to do data recovery?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1040022460
@szetszwo I want to discuss a question about ozone BCSID.
Does BCSID still make sense when you use stream to transfer data and put A Block into the stream process? Because raft transport is deprecated, playback of raft log is eliminated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042604015
@szetszwo https://docs.google.com/document/d/1mS3GqovQ3D1b7V0L3--VF9xhl5jdId1mSL0cQNb7uHo/edit
This is the process of testing reports and locating problems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042656987
@szetszwo
This is the process of testing reports and locating problems
https://docs.google.com/document/d/1mS3GqovQ3D1b7V0L3--VF9xhl5jdId1mSL0cQNb7uHo/edit
Can you take a look at our test report and consider further optimizing the performance of ratis stream over ozone based on the problems identified in the test report.
Discuss how to optimize the scheme.
@captainzmc and I can participate in the development of optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky edited a comment on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky edited a comment on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1040022460
@szetszwo I want to discuss a question about ozone BCSID.
Does BCSID still make sense when you use stream to transfer data and put A Block into the stream process? Because raft transport is deprecated, playback of raft log is eliminated.
I can open a JIRA in Ozone and discuss this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1041407676
> ... could we get a BCSID in another way than raft log?
We need a BCSID and also the ability to commit transactions. I guess there are no easy ways. Otherwise, we can use it to replace the Raft Consensus Algorithm in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042653639
> > We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.
>
> This is not really a fix since it changes the functionality.
>
> ```
> commit f349fdf2488e254d0fa2eeb17bbaf44de2b0932d
> Author: hao guo <gu...@360.cn>
> Date: Wed Dec 15 10:29:13 2021 +0800
>
> RATIS-1438. Add request timeout to ratis Streaming (#563)
> ```
>
> "TimeoutIOException: Timeout 3000ms" probably started happening after RATIS-1438. How about we increase the timeout value, say to 10 seconds?
> DataStreamManagement#startTransaction
It can be changed to 10 seconds and I will submit a new PR for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042634240
> We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.
This is not really a fix since it changes the functionality.
```
commit f349fdf2488e254d0fa2eeb17bbaf44de2b0932d
Author: hao guo <gu...@360.cn>
Date: Wed Dec 15 10:29:13 2021 +0800
RATIS-1438. Add request timeout to ratis Streaming (#563)
```
"TimeoutIOException: Timeout 3000ms" probably started happening after RATIS-1438. How about we increase the timeout value, say to 10 seconds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042537242
> ..., generating an ID on the Primary node ...
How to make sure the ID is unique? All the nodes could be the Primary node at some point of time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1042547408
> https://issues.apache.org/jira/browse/RATIS-1513
@guohao-rosicky, @captainzmc, I really hope that we could fix "TimeoutIOException: Timeout 3000ms". How about we fix it first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1039969825
@szetszwo What do you think of this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] guohao-rosicky commented on pull request #601: RATIS-1524. Optional DataStreamManagement#startTransaction configuration
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #601:
URL: https://github.com/apache/ratis/pull/601#issuecomment-1041315492
Thank you @szetszwo , I have understood BCSID.
In one of my test versions of ozone, I generated raft logs by sending raft async RPC through DataStreamManagement#startTransaction to get the BCSID with the raft log index of ozone, Throughput is very small because raft needs to sort.
In another version of my test, if DataStreamManagement#startTransaction throughput was doubled by skipping it, could we get a BCSID in another way than raft log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org