You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ratis.apache.org by ms...@apache.org on 2019/07/01 15:26:27 UTC
[incubator-ratis] branch ratis-ozone updated (fe2b15d -> 2337318)
This is an automated email from the ASF dual-hosted git repository.
msingh pushed a change to branch ratis-ozone
in repository https://gitbox.apache.org/repos/asf/incubator-ratis.git.
from fe2b15d RATIS-532. UnorderedAsync requests should fail instead of retrying the request in case of GroupMismatchException. Contributed by Shashikant Banerjee.
add 1698e73 RATIS-523. Switch chatty debug to trace logging
add 4a1697a RATIS-524. Mark meta-quorum jcommander options as required
add cbdf7bd RATIS-525. Usability improvements to VerificationTool
add 912365d RATIS-526. Push building the docker image and launching the client env into scripts
add f76d779 RATIS-527. Basic script wrappers around common logservice commands
add c01c5e9 RATIS-533. Set better default Raft properties from Ratis
add 2e7211f RATIS-497. IllegalStateException: ILLEGAL TRANSITION: In LogAppender(s4 -> s1), CLOSING -> RUNNING. Contributed by Tsz Wo Nicholas Sze.
add e157ce5 RATIS-518. Add request specific retry policy support.
add c7b35c3 RATIS-529. Leader should limit pending requests to avoid OOM. Contributed by Tsz Wo Nicholas Sze.
add 21d7526 RATIS-515. IllegalStateException while updating matchIndex. Contributed by Tsz Wo Nicholas Sze.
add 3419793 RATIS-555. Move RaftLog and the implementations to separated packages.
add 5f59dea RATIS-546. Configurable value sizes on VerificationTool
add 21c9798 RATIS-534. Warn that the netty useCacheForAllThreads feature may cause JVM GC pauses. Contributed by Lokesh Jain
add d5cd3f6 RATIS-542. When a server restarts, purge(-1) causes IllegalStateException.
add de557a7 RATIS-558. GrpcLogAppender doesn't reset the log entries to follower on reply to inconsistent entries.
add e493b60 RATIS-559. LogAppender.getPrevious throws IllegalStateException.
add 346c024 RATIS-563. Purge logs in takeSnapshot only after all the nodes have caught up with the index. Contributed by Tsz Wo Nicholas Sze.
add 4b613c1 RATIS-568. Server sliding window should fail out of order requests on close. Contributed by Tsz Wo Nicholas Sze.
add ee1e3a5 RATIS-566. OrderedStreamObservers#closeAllExisting should close the streams for a certain group id only. Contributed by Tsz Wo Nicholas Sze.
add 231b35c RATIS-565. Move the ordered async client code to a new class. Contributed by Tsz Wo Nicholas Sze.
add b98fac8 RATIS-549 RATIS-550 RATIS-552 RATIS-553 Metric Framework for Ratis
add ad6949f RATIS-520: LogService Configuration
add c31cf5c RATIS-545. Leader Election timeout should consider JVM pause interval. Contributed by Lokesh Jain
add 4ee3438 RATIS-167. Implement a high performance OutputStream using async API.
add be131d6 RATIS-480. Update scripts to work in both dev checkouts and binary tarball
add 8a7fac9 RATIS-571. Client may send first request in sliding window with firstFlag as false. Contributed by Lokesh Jain.
add 971c970 RATIS-574. Client should not use configured sleep interval in RetryPolicy when it gets NotLeaderException. Contributed by Siddharth Wagle
add 300d9c5 RATIS-564. Add raft group Id to install snapshot notification. Contributed by Siddharth Wagle.
add 008c009 RATIS-577. NPE in LogAppender while creating a new request.
add 3c41936 RATIS-579. ExitUtils.terminate results in JVM terminated for peers not part of the group. Contributed by Siddharth Wagle
add f7a394c RATIS-587. FinalizeLogSegment should flush the contents of log.
add be68f84 RATIS-597. Add notifyLeader API in state machine. Contributed by Ankit Singhal
add 6d7afec RATIS-601. Fix NotLeaderException handling. Contributed by Mukul Kumar Singh
add 9d7c7a3 RATIS-573. Handle Raft Log Append Failure. Contributed by Supratim Deka
add 5dc6eee RATIS-590. Deadlock in ratis client. Contributed by Lokesh Jain
add e08f2d1 Revert "RATIS-601. Fix NotLeaderException handling. Contributed by Mukul Kumar Singh"
add 051fe4e RATIS-595. appendEntry future should be completed only after the entry is flushed.
add 937f68d RATIS-601. Fix NotLeaderException handling. Contributed by Mukul Kumar Singh.
add 43ab53c RATIS-575. TestWatchRequestWithGrpc#testWatchRequestTimeout fails with NotReplicatedException. Contributed by Tsz Wo Nicholas Sze.
add 5f778e2 RATIS-592. One node ratis writes fail forever after first NotLeaderException or LeaderNotReadyException. Contributed by Siddharth Wagle
add 2337318 RATIS-604. OrderedRequestStreamObserver does not cleanup when responding error. Contributed by Lokesh Jain
No new revisions were added by this update.
Summary of changes:
pom.xml | 15 +-
ratis-assembly/pom.xml | 4 +
ratis-assembly/src/main/assembly/bin.xml | 23 +-
.../org/apache/ratis/client/RaftClientRpc.java | 12 +-
.../apache/ratis/client/impl/ClientProtoUtils.java | 27 +-
.../org/apache/ratis/client/impl/OrderedAsync.java | 253 ++++++++++++++++++
.../apache/ratis/client/impl/RaftClientImpl.java | 286 ++++++---------------
.../ratis/client/impl/RaftClientTestUtil.java | 5 +-
.../apache/ratis/client/impl/RaftOutputStream.java | 136 ++++++++++
.../apache/ratis/client/impl/UnorderedAsync.java | 24 +-
ratis-common/pom.xml | 5 -
.../java/org/apache/ratis/conf/RaftProperties.java | 5 +
.../ratis/protocol/LeaderNotReadyException.java | 9 +-
.../apache/ratis/protocol/NotLeaderException.java | 17 +-
.../org/apache/ratis/protocol/RaftClientReply.java | 23 +-
.../apache/ratis/protocol/RaftClientRequest.java | 2 +-
.../ratis/protocol/RaftRetryFailureException.java | 12 +-
.../ratis/protocol/SetConfigurationRequest.java | 16 +-
.../ResourceUnavailableException.java} | 12 +-
.../java/org/apache/ratis/retry/RetryPolicies.java | 123 ++++++---
.../java/org/apache/ratis/retry/RetryPolicy.java | 16 +-
.../java/org/apache/ratis/util/AtomicUtils.java | 60 +++++
.../org/apache/ratis/util/CollectionUtils.java | 21 ++
.../main/java/org/apache/ratis/util/DataQueue.java | 7 +-
.../main/java/org/apache/ratis/util/ExitUtils.java | 2 +-
.../main/java/org/apache/ratis/util/JavaUtils.java | 21 +-
.../java/org/apache/ratis/util/Preconditions.java | 17 ++
.../java/org/apache/ratis/util/ProtoUtils.java | 10 +-
.../org/apache/ratis/util/ResourceSemaphore.java | 51 ++--
.../java/org/apache/ratis/util/SlidingWindow.java | 87 ++++---
.../java/org/apache/ratis/util/TimeDuration.java | 20 +-
.../src/test/java/org/apache/ratis/BaseTest.java | 12 +
ratis-examples/pom.xml | 8 -
ratis-examples/src/main/bin/client.sh | 16 +-
ratis-examples/src/main/bin/common.sh | 11 +-
ratis-examples/src/main/bin/server.sh | 2 +-
ratis-examples/src/main/bin/start-all.sh | 28 +-
.../java/org/apache/ratis/grpc/GrpcFactory.java | 22 ++
.../main/java/org/apache/ratis/grpc/GrpcUtil.java | 7 +-
.../grpc/client/GrpcClientProtocolClient.java | 19 +-
.../grpc/client/GrpcClientProtocolService.java | 81 ++++--
.../apache/ratis/grpc/client/GrpcClientRpc.java | 13 +-
.../ratis/grpc/client/GrpcClientStreamer.java | 8 +-
.../apache/ratis/grpc/client/GrpcOutputStream.java | 4 +-
.../apache/ratis/grpc/server/GrpcLogAppender.java | 134 ++++------
.../org/apache/ratis/grpc/server/GrpcService.java | 5 +-
ratis-logservice/README.md | 20 +-
ratis-logservice/TUNING.md | 58 +++++
ratis-logservice/{Dockerfile => build-docker.sh} | 28 +-
.../resources/log4j.properties => client-env.sh} | 10 +-
ratis-logservice/logservice-example.xml | 88 +++++++
ratis-logservice/src/assembly/assembly.xml | 7 +
.../src/assembly/bin/load-test | 12 +-
.../src/assembly/bin/shell | 12 +-
.../logservice/api/LogServiceConfiguration.java | 188 +++++++++++---
.../ratis/logservice/client/LogServiceClient.java | 4 +-
.../apache/ratis/logservice/common/Constants.java | 29 ++-
.../ratis/logservice/impl/LogStreamImpl.java | 2 +-
.../ratis/logservice/impl/LogWriterImpl.java | 6 +-
.../metrics/LogServiceMetricsRegistry.java | 69 +++++
.../apache/ratis/logservice/server/BaseServer.java | 59 ++++-
.../apache/ratis/logservice/server/LogServer.java | 45 +++-
.../logservice/server/LogServiceRaftLogReader.java | 16 +-
.../ratis/logservice/server/LogStateMachine.java | 101 ++++++--
.../ratis/logservice/server/MetaStateMachine.java | 58 +++--
.../ratis/logservice/server/MetadataServer.java | 26 +-
.../apache/ratis/logservice/server/ServerOpts.java | 29 ++-
.../logservice/shell/LogServiceShellOpts.java | 2 +-
.../ratis/logservice/tool/VerificationTool.java | 220 ++++++++++++----
.../ratis/logservice/LogServiceReadWriteBase.java | 68 ++++-
.../ratis/logservice/server/TestBaseServer.java | 49 ++++
.../ratis/logservice/server/TestMetaServer.java | 62 ++++-
.../logservice/tool/TestVerificationTool.java | 59 +++++
{ratis-server => ratis-metrics}/pom.xml | 44 ++--
.../org/apache/ratis/metrics/MetricRegistries.java | 86 +++++++
.../ratis/metrics/MetricRegistriesLoader.java | 95 +++++++
.../ratis/metrics/MetricRegistryFactory.java | 29 ++-
.../apache/ratis/metrics/MetricRegistryInfo.java | 87 +++++++
.../ratis/metrics/impl/MetricRegistriesImpl.java | 76 ++++++
.../metrics/impl/MetricRegistryFactoryImpl.java | 25 +-
.../ratis/metrics/impl/RatisMetricRegistry.java | 103 ++++++++
.../apache/ratis/metrics/impl/RefCountingMap.java | 86 +++++++
...rg.apache.hadoop.hbase.metrics.MetricRegistries | 11 +-
.../ratis/metrics/TestMetricRegistriesLoader.java | 54 ++++
.../ratis/metrics/impl/TestRefCountingMap.java | 147 +++++++++++
ratis-proto/src/main/proto/Raft.proto | 5 +
ratis-server/pom.xml | 10 +
.../apache/ratis/server/RaftServerConfigKeys.java | 87 +++++--
.../org/apache/ratis/server/RaftServerRpc.java | 3 +-
.../org/apache/ratis/server/impl/FollowerInfo.java | 18 +-
.../apache/ratis/server/impl/FollowerState.java | 7 +-
.../org/apache/ratis/server/impl/LeaderState.java | 83 +++---
.../org/apache/ratis/server/impl/LogAppender.java | 183 ++++++++-----
.../apache/ratis/server/impl/PendingRequests.java | 92 +++++--
.../ratis/server/impl/RaftConfiguration.java | 27 +-
.../ratis/server/impl/RaftServerConstants.java | 6 +-
.../apache/ratis/server/impl/RaftServerImpl.java | 143 +++++------
.../apache/ratis/server/impl/ServerProtoUtils.java | 22 +-
.../org/apache/ratis/server/impl/ServerState.java | 54 ++--
.../ratis/server/impl/StateMachineUpdater.java | 212 ++++++++-------
.../apache/ratis/server/impl/WatchRequests.java | 45 ++--
.../apache/ratis/server/metrics/RatisMetrics.java | 51 ++++
.../ratis/server/{storage => raftlog}/RaftLog.java | 79 ++++--
.../{storage => raftlog}/RaftLogIOException.java | 4 +-
.../server/{storage => raftlog}/RaftLogIndex.java | 9 +-
.../{storage => raftlog}/RaftLogSequentialOps.java | 2 +-
.../{storage => raftlog/memory}/MemoryRaftLog.java | 24 +-
.../segmented}/BufferedChannelBase.java | 4 +-
.../segmented}/BufferedWriteChannel.java | 82 ++----
.../segmented}/CacheInvalidationPolicy.java | 6 +-
.../{storage => raftlog/segmented}/LogSegment.java | 8 +-
.../segmented}/SegmentedRaftLog.java | 74 +++---
.../segmented/SegmentedRaftLogCache.java} | 41 ++-
.../segmented}/SegmentedRaftLogFormat.java | 2 +-
.../segmented/SegmentedRaftLogInputStream.java} | 23 +-
.../segmented/SegmentedRaftLogOutputStream.java} | 17 +-
.../segmented/SegmentedRaftLogReader.java} | 14 +-
.../segmented/SegmentedRaftLogWorker.java} | 189 ++++++++++----
.../org/apache/ratis/server/storage/MetaFile.java | 8 +-
.../apache/ratis/server/storage/RaftStorage.java | 24 +-
.../ratis/server/storage/RaftStorageDirectory.java | 6 +-
.../apache/ratis/statemachine/StateMachine.java | 29 ++-
.../ratis/statemachine/impl/BaseStateMachine.java | 16 ++
.../java/org/apache/ratis/LogAppenderTests.java | 2 +-
.../java/org/apache/ratis/MiniRaftCluster.java | 34 +--
.../org/apache/ratis/OutputStreamBaseTest.java | 168 ++++++------
.../test/java/org/apache/ratis/RaftAsyncTests.java | 41 ++-
.../test/java/org/apache/ratis/RaftBasicTests.java | 2 +-
.../org/apache/ratis/RaftExceptionBaseTest.java | 8 +-
.../test/java/org/apache/ratis/RaftTestUtil.java | 2 +-
.../apache/ratis/RequestLimitAsyncBaseTest.java | 124 +++++++++
.../java/org/apache/ratis/RetryCacheTests.java | 4 +-
.../java/org/apache/ratis/WatchRequestTests.java | 38 ++-
.../apache/ratis/server/ServerRestartTests.java | 6 +-
.../ratis/server/impl/LeaderElectionTests.java | 6 +-
.../server/impl/RaftReconfigurationBaseTest.java | 13 +-
.../ratis/server/impl/RaftServerTestUtil.java | 11 +-
.../impl/RaftStateMachineExceptionTests.java | 2 +-
.../segmented/SegmentedRaftLogTestUtils.java | 19 +-
.../ratis/server/storage/RaftStorageTestUtils.java | 15 +-
.../ratis/statemachine/RaftSnapshotBaseTest.java | 6 +-
.../statemachine/SimpleStateMachine4Testing.java | 79 +++++-
.../java/org/apache/ratis/TestRetryPolicy.java | 80 +++++-
...tionWithGrpc.java => TestGrpcOutputStream.java} | 31 ++-
.../ratis/grpc/TestInstallSnapshotWithGrpc.java | 10 +-
...Grpc.java => TestRaftOutputStreamWithGrpc.java} | 19 +-
.../apache/ratis/grpc/TestRaftServerWithGrpc.java | 1 +
.../org/apache/ratis/grpc/TestRaftWithGrpc.java | 2 +-
...rpc.java => TestRequestLimitAsyncWithGrpc.java} | 6 +-
.../{storage => raftlog}/TestRaftLogIndex.java | 2 +-
.../server/{ => raftlog}/TestRaftLogMetrics.java | 40 +--
.../segmented}/TestCacheEviction.java | 9 +-
.../segmented/TestLogSegment.java} | 28 +-
.../segmented}/TestRaftLogReadWrite.java | 38 ++-
.../segmented}/TestSegmentedRaftLog.java | 122 +++++----
.../segmented/TestSegmentedRaftLogCache.java} | 13 +-
.../apache/ratis/util/TestDataBlockingQueue.java | 3 +-
.../org/apache/ratis/util/TestPreconditions.java | 53 ++++
158 files changed, 4807 insertions(+), 1800 deletions(-)
create mode 100644 ratis-client/src/main/java/org/apache/ratis/client/impl/OrderedAsync.java
create mode 100644 ratis-client/src/main/java/org/apache/ratis/client/impl/RaftOutputStream.java
copy ratis-common/src/main/java/org/apache/ratis/protocol/{StaleReadException.java => exceptions/ResourceUnavailableException.java} (76%)
create mode 100644 ratis-common/src/main/java/org/apache/ratis/util/AtomicUtils.java
copy ratis-server/src/main/java/org/apache/ratis/server/storage/FileInfo.java => ratis-common/src/main/java/org/apache/ratis/util/ResourceSemaphore.java (50%)
create mode 100644 ratis-logservice/TUNING.md
copy ratis-logservice/{Dockerfile => build-docker.sh} (59%)
mode change 100644 => 100755
copy ratis-logservice/{src/main/resources/log4j.properties => client-env.sh} (73%)
mode change 100644 => 100755
create mode 100644 ratis-logservice/logservice-example.xml
copy ratis-examples/src/main/resources/log4j.properties => ratis-logservice/src/assembly/bin/load-test (73%)
mode change 100644 => 100755
copy ratis-examples/src/main/resources/log4j.properties => ratis-logservice/src/assembly/bin/shell (73%)
mode change 100644 => 100755
create mode 100644 ratis-logservice/src/main/java/org/apache/ratis/logservice/metrics/LogServiceMetricsRegistry.java
create mode 100644 ratis-logservice/src/test/java/org/apache/ratis/logservice/server/TestBaseServer.java
create mode 100644 ratis-logservice/src/test/java/org/apache/ratis/logservice/tool/TestVerificationTool.java
copy {ratis-server => ratis-metrics}/pom.xml (78%)
create mode 100644 ratis-metrics/src/main/java/org/apache/ratis/metrics/MetricRegistries.java
create mode 100644 ratis-metrics/src/main/java/org/apache/ratis/metrics/MetricRegistriesLoader.java
copy ratis-common/src/main/java/org/apache/ratis/metrics/RatisMetricsRegistry.java => ratis-metrics/src/main/java/org/apache/ratis/metrics/MetricRegistryFactory.java (66%)
create mode 100644 ratis-metrics/src/main/java/org/apache/ratis/metrics/MetricRegistryInfo.java
create mode 100644 ratis-metrics/src/main/java/org/apache/ratis/metrics/impl/MetricRegistriesImpl.java
rename ratis-common/src/main/java/org/apache/ratis/metrics/RatisMetricsRegistry.java => ratis-metrics/src/main/java/org/apache/ratis/metrics/impl/MetricRegistryFactoryImpl.java (69%)
create mode 100644 ratis-metrics/src/main/java/org/apache/ratis/metrics/impl/RatisMetricRegistry.java
create mode 100644 ratis-metrics/src/main/java/org/apache/ratis/metrics/impl/RefCountingMap.java
copy ratis-examples/src/main/resources/log4j.properties => ratis-metrics/src/main/resources/META-INF/services/org.apache.hadoop.hbase.metrics.MetricRegistries (73%)
create mode 100644 ratis-metrics/src/test/java/org/apache/ratis/metrics/TestMetricRegistriesLoader.java
create mode 100644 ratis-metrics/src/test/java/org/apache/ratis/metrics/impl/TestRefCountingMap.java
create mode 100644 ratis-server/src/main/java/org/apache/ratis/server/metrics/RatisMetrics.java
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog}/RaftLog.java (85%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog}/RaftLogIOException.java (96%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog}/RaftLogIndex.java (90%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog}/RaftLogSequentialOps.java (99%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/memory}/MemoryRaftLog.java (92%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/segmented}/BufferedChannelBase.java (96%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/segmented}/BufferedWriteChannel.java (55%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/segmented}/CacheInvalidationPolicy.java (97%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/segmented}/LogSegment.java (97%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/segmented}/SegmentedRaftLog.java (87%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage/RaftLogCache.java => raftlog/segmented/SegmentedRaftLogCache.java} (93%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage => raftlog/segmented}/SegmentedRaftLogFormat.java (98%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage/LogInputStream.java => raftlog/segmented/SegmentedRaftLogInputStream.java} (90%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage/LogOutputStream.java => raftlog/segmented/SegmentedRaftLogOutputStream.java} (92%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage/LogReader.java => raftlog/segmented/SegmentedRaftLogReader.java} (96%)
rename ratis-server/src/main/java/org/apache/ratis/server/{storage/RaftLogWorker.java => raftlog/segmented/SegmentedRaftLogWorker.java} (75%)
rename ratis-test/src/test/java/org/apache/ratis/grpc/TestRaftStream.java => ratis-server/src/test/java/org/apache/ratis/OutputStreamBaseTest.java (64%)
create mode 100644 ratis-server/src/test/java/org/apache/ratis/RequestLimitAsyncBaseTest.java
copy ratis-test/src/test/java/org/apache/ratis/grpc/TestRaftReconfigurationWithGrpc.java => ratis-server/src/test/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogTestUtils.java (68%)
copy ratis-test/src/test/java/org/apache/ratis/grpc/{TestRaftReconfigurationWithGrpc.java => TestGrpcOutputStream.java} (50%)
copy ratis-test/src/test/java/org/apache/ratis/grpc/{TestRaftReconfigurationWithGrpc.java => TestRaftOutputStreamWithGrpc.java} (72%)
copy ratis-test/src/test/java/org/apache/ratis/grpc/{TestRaftAsyncExceptionWithGrpc.java => TestRequestLimitAsyncWithGrpc.java} (85%)
rename ratis-test/src/test/java/org/apache/ratis/server/{storage => raftlog}/TestRaftLogIndex.java (98%)
rename ratis-test/src/test/java/org/apache/ratis/server/{ => raftlog}/TestRaftLogMetrics.java (78%)
rename ratis-test/src/test/java/org/apache/ratis/server/{storage => raftlog/segmented}/TestCacheEviction.java (95%)
rename ratis-test/src/test/java/org/apache/ratis/server/{storage/TestRaftLogSegment.java => raftlog/segmented/TestLogSegment.java} (93%)
rename ratis-test/src/test/java/org/apache/ratis/server/{storage => raftlog/segmented}/TestRaftLogReadWrite.java (87%)
rename ratis-test/src/test/java/org/apache/ratis/server/{storage => raftlog/segmented}/TestSegmentedRaftLog.java (84%)
rename ratis-test/src/test/java/org/apache/ratis/server/{storage/TestRaftLogCache.java => raftlog/segmented/TestSegmentedRaftLogCache.java} (97%)
create mode 100644 ratis-test/src/test/java/org/apache/ratis/util/TestPreconditions.java