You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by el...@apache.org on 2021/06/02 13:06:37 UTC
[ozone] branch HDDS-3816-ec updated (ad790b6 -> f904539)
This is an automated email from the ASF dual-hosted git repository.
elek pushed a change to branch HDDS-3816-ec
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from ad790b6 HDDS-4883. Persist replicationIndex on datanode side (#2069)
add 6f9c3f2 HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (#2187)
add bddc873 HDDS-4515. Datanodes should be able to persist and load CRL (#2181)
add ce29843 HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (#2184)
add bfb6be8 HDDS-5152. Fix Suggested leader in Client. (#2189)
add bccc439 HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (#2188)
add a920f25 HDDS-5153. Decommissioning a dead node should complete immediately (#2190)
add 66a411b HDDS-4585. Support bucket acl operation in S3g (#1701)
add 5dd0943 HDDS-5144. Create github check to alert when dependency tree is changed (#2177)
add 10312fb HDDS-4983. Display key offset for each block in command key info (#2051)
add 799c38e Revert "HDDS-5144. Create github check to alert when dependency tree is changed (#2177)"
add dcac0f1 HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (#2195)
add 48b84c5 HDDS-2212. Genconf tool should generate config files for secure clust… (#1788)
add a3e7f04 HDDS-5182. Acceptance test may exit with 0 in case of error (#2212)
add 2e07c9f HDDS-5178. Update project information of Contribution guideline (#2210)
add 8bbbf3f HDDS-5177. Update link of weekly calls in README (#2209)
add 00eba3e HDDS-5144. Create github check to alert when dependency tree is changed (#2199)
add 142a2dd HDDS-5185. Update commons-io to 2.8.0 (#2215)
add d430ff3 HDDS-5159. Make periodic disk check interval configurable. (#2193)
add 74496a7 HDDS-5196. Fix typo "vaild" in HeartbeatEndpointTask.java and RegisterEndpointTask (#2222).
add 2061d7d HDDS-5192. Intermittent failure in TestOzoneRpcClient due to volume name conflict (#2220)
add 5e8ae99 HDDS-5186. If primordial SCM id is set, a non-HA cluster can not be initialized. (#2218)
add a16e648 HDDS-5126.Recon should check new containers of a container report with batch (#2172)
add 2c37a15 HDDS-5183. For AccessControlException do not perform failover. (#2213)
add 75c5c10 HDDS-5099. Error with unit test for hdds.container-service TestSchemaOneBackwardsCompatibility (#2183)
add 8e8a01d HDDS-5197. Pass option variables to OZONE_OPTS before adding default GC opts (#2225)
add 7ba1499 HDDS-5198. Remove JBoss repo definition (#2226)
add 2c4a4de HDDS-5173. Divide snapshot related work into notifyInstallSnapshotFromLeader and reinitialize for SCMStateMachine. (#2224)
add 49538aa HDDS-5207. Intermittent failure in TestRatisPipelineProvider#testCreatePipelineWithFactorThree (#2233)
add 3e66778 HDDS-5200. Fix scm roles command if one of the host is unresolvable. (#2227)
add abe136f HDDS-5203. Allow suppressing deprecation warning for HADOOP_ variables (#2231)
add 248d72d Bump logical release name of Ozone 1.2 (#2238)
add b5cbb65 HDDS-5164. Improve client and server logging. (#2234)
add 7262004 HDDS-5089. On-demand disk checker for hdds volume (#2158)
add 842e6f6 HDDS-4729. Add token support for container admin operations (#2186)
add 03620da HDDS-5059. Increase number of client retries/ failovers to OMs (#2171)
add f8a06e0 HDDS-5202. Use scm#checkLeader before processing client requests . (#2229)
add 6075acb HDDS-5205. Make admin check work for SCM HA cluster (#2236)
add 0f6bb36 HDDS-5040. Append om1,om2,om3 to ozone.administrators config for secure OM HA docker config (#2245)
add a7c8695 HDDS-5160. Remove some Freon integration tests (#2194)
add 6b24c18 HDDS-4976. Add container replica related commands to debug (#2070)
add fd71f09 The modificationTime should modify when setAcl success (#2242)
add 4189088 HDDS-5165. OM DB checkpoint servlet not accessible in a secure cluster (#2200)
add eff4914 HDDS-5208. bump rocksdb version to 6.20.3 (#2235)
add 36778af HDDS-5128. Return specific error messages for S3-SDK (#2169)
add 19f989f HDDS-5176. Increase default block cache capacity for Datanodes (#2207)
add 9cab560 HDDS-5155. Add revokeCertificate to SCMSecurityProtocolServer. (#2191)
add 727b258 HDDS-5189. fix markdown files to make them hugo83 compatible (and valid) (#2232)
add 7a4a93e HDDS-5229. Update copyright year in NOTICE (#2248)
add ac7fc7e HDDS-5082. Create unit (!) test for OzoneClient (#2138)
add 2254abf HDDS-5216. Fix race condition causing SCM failOverProxy which is causing failover wrongly. (#2247)
add fcc8e92 HDDS-5187. Avoid Maven connection errors in CI (#2237)
add 10a5b39 HDDS-5113. Disable flaky TestBlockDeletingService#testContainerThrottle
add 91993c1 HDDS-5234.Change default grpc and ratis ports for scm ha (#2251)
add ea1ed7c HDDS-5238. Disable animal-sniffer maven plugin (#2253)
add 9f6d7f2 HDDS-5167. Add clean option for Freon randomkeys to clean generated objects (#2198)
add d63dd6b HDDS-5239. Create http Basedir if not exist (#2255)
add 5a50866 HDDS-5209. Datanode hasEnoughSpace check should apply on volume instead of global DN (#2246)
add 30fe6ea HDDS-5245. Fix OzoneContainer TLS configuration. (#2264)
add 4fd8187 HDDS-5206. Support revoking S3 secret (#2239)
add 08375d7 HDDS-5246. Wait for ever to obtain CA list which is needed during OM/DN startup (#2266)
add 70ef475 HDDS-5233. SCM subsequent init failed when previous scm init failed. (#2249)
add 09c2278 HDDS-5257. Avoid SCM call to get CA certs in non-HA from OM. (#2273)
add e1acbb0 HDDS-5256. Fix fall back of config in SCM HA Cluster (#2272)
add 61a237a HDDS-5261. Delete Useless ozone subcommand (#2275)
add e50fe52 HDDS-5041. Use getShortUserName in getTrashRoot(s) (#2244)
add fc7b0bd HDDS-5260. Bump node to v16.2.0 for Recon (#2276)
add 9080fc3 HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log. (#2267)
add e00fa80 HDDS-5206. Addendum: Support revoking S3 secret (#2270)
add 8d29be1 HDDS-5142. Make generic streaming client/service for container re-replication, data read, scm/om snapshot download (#2256)
add ab8f07d HDDS-5249. Race Condition between Full and Incremental Container Reports (#2268)
add 931a041 HDDS-5266 Misspelt words in S3MultipartUploadCommitPartRequest.java line 202 (#2279)
add 355096b HDDS-5236. Require block token for more operations (#2254)
add 19f5bb2 HDDS-5250. Build integration tests with Maven cache (#2269)
add 0b4779c HDDS-5073. Use ReplicationConfig on client side (#2136)
new 9c4008d Merge remote-tracking branch 'origin/master' into HDDS-3816-ec
new c960ec7 Merge remote-tracking branch 'origin/HDDS-3816-ec' into HDDS-3816-ec
add 2dbf04e HDDS-5191. Increase default pvc storage size (#2219)
add 000e7c5 HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space. (#2271)
add 2671b48 HDDS-5219. Limit number of bad volumes by dfs.datanode.failed.volumes.tolerated. (#2243)
add d9ae1b3 HDDS-5241. SCM UI should have leader/follower and Primordial SCM information (#2260)
add 53bbabf HDDS-5275. Datanode Report Publisher publishes one extra report after DN shutdown (#2283)
add 4301315 HDDS-5158. Add documentation for SCM HA Security. (#2205)
add d567360 HDDS-5273. Handle unsecure cluster convert to secure cluster for SCM. (#2281)
add 00b675d HDDS-4927. Determine over and under utilized datanodes in Container Balancer. (#2230)
add 8586815 HDDS-4043. allow deletion from Trash directory without -skipTrash option (#2110)
add fa4ddbc HDDS-4936. Change ozone groupId from org.apache.hadoop to org.apache.ozone (#2018)
add 3ffbdd0 HDDS-4993. Add guardrail for reserved buffer size when DN reads a chunk (#2058)
add e5a774f HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests (#2165)
add 5fd9780 HDDS-5272. Make ozonefs.robot execution repeatable (#2280)
add 12e2918 HDDS-5280. Make XceiverClientManager creation when necessary in ContainerOperationClient (#2289)
add 6b09219 HDDS-5264. SCM should send token for CloseContainer command (#2277)
add e35dde9 HDDS-5151. Support ByteBuffer read in OzoneInputStream (#2203)
add 96d9ed7 HDDS-5240. Fix out of bound exception when loading auditparser (#2258)
add 69b92a0 HDDS-5281. Add reinitialize() for SequenceIdGenerator. (#2292)
new f904539 Merge remote-tracking branch 'origin/master' into HDDS-3816-ec
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/buildenv/Dockerfile | 20 --
.github/buildenv/entrypoint.sh | 18 --
.github/workflows/post-commit.yml | 103 ++++---
hadoop-hdds/client/pom.xml | 30 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 29 +-
.../hadoop/hdds/scm/storage/ByteArrayReader.java | 67 ++++
.../hadoop/hdds/scm/storage/ByteBufferReader.java | 71 +++++
.../hdds/scm/storage/ByteReaderStrategy.java | 41 +++
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 38 ++-
.../hdds/scm/storage/TestBlockInputStream.java | 35 +++
hadoop-hdds/common/pom.xml | 26 +-
.../apache/hadoop/hdds/DFSConfigKeysLegacy.java | 5 -
.../java/org/apache/hadoop/hdds/HddsUtils.java | 17 +
.../hadoop/hdds/client/RatisReplicationConfig.java | 15 +
.../hadoop/hdds/client/ReplicationConfig.java | 55 ++++
.../hdds/client/StandaloneReplicationConfig.java | 9 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 4 +-
.../hadoop/hdds/client/TestReplicationConfig.java | 147 +++++++++
hadoop-hdds/config/pom.xml | 10 +-
hadoop-hdds/container-service/pom.xml | 42 +--
.../container/common/report/ReportPublisher.java | 4 +-
.../common/statemachine/DatanodeConfiguration.java | 30 ++
.../common/statemachine/DatanodeStateMachine.java | 6 +
.../CloseContainerCommandHandler.java | 9 +-
.../states/endpoint/HeartbeatEndpointTask.java | 4 +
.../container/common/volume/HddsVolumeChecker.java | 11 +-
.../container/common/volume/MutableVolumeSet.java | 42 ++-
.../container/keyvalue/helpers/ChunkUtils.java | 12 +
.../keyvalue/impl/ChunkManagerDummyImpl.java | 6 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 5 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 2 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 5 +-
.../hadoop/ozone/protocol/commands/SCMCommand.java | 10 +
.../container/common/TestBlockDeletingService.java | 2 +-
.../TestSchemaOneBackwardsCompatibility.java | 4 +-
.../common/impl/TestContainerPersistence.java | 2 +-
.../container/common/impl/TestHddsDispatcher.java | 5 +-
.../common/report/TestReportPublisher.java | 9 +-
.../statemachine/TestDatanodeConfiguration.java | 12 +
.../container/common/volume/TestVolumeSet.java | 5 +-
.../common/volume/TestVolumeSetDiskChecks.java | 11 +-
.../keyvalue/TestKeyValueBlockIterator.java | 2 +-
.../keyvalue/TestKeyValueContainerCheck.java | 2 +-
.../container/keyvalue/TestKeyValueHandler.java | 3 +-
.../keyvalue/impl/CommonChunkManagerTestCases.java | 33 ++
.../container/ozoneimpl/TestContainerReader.java | 2 +-
.../container/ozoneimpl/TestOzoneContainer.java | 3 +-
hadoop-hdds/docs/content/feature/SCM-HA.md | 68 +++-
hadoop-hdds/docs/content/feature/scm-secure-ha.png | Bin 0 -> 46757 bytes
hadoop-hdds/docs/content/interface/O3fs.md | 4 +-
hadoop-hdds/docs/content/interface/O3fs.zh.md | 8 +-
hadoop-hdds/docs/content/interface/Ofs.md | 4 +-
hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md | 6 +-
.../docs/content/recipe/SparkOzoneFSK8S.zh.md | 6 +-
hadoop-hdds/docs/pom.xml | 6 +-
hadoop-hdds/framework/pom.xml | 34 +-
...etManager.java => ContainerTokenGenerator.java} | 58 ++--
.../token/ContainerTokenSecretManager.java | 30 +-
.../hdds/scm/protocol/TestReplicationConfig.java | 86 ------
hadoop-hdds/hadoop-dependency-client/pom.xml | 6 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 6 +-
hadoop-hdds/hadoop-dependency-test/pom.xml | 6 +-
hadoop-hdds/interface-admin/pom.xml | 10 +-
hadoop-hdds/interface-client/pom.xml | 6 +-
hadoop-hdds/interface-server/pom.xml | 10 +-
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 1 +
hadoop-hdds/pom.xml | 78 ++---
hadoop-hdds/server-scm/pom.xml | 56 ++--
.../scm/container/CloseContainerEventHandler.java | 9 +
.../hdds/scm/container/ReplicationManager.java | 12 +-
.../scm/container/balancer/ContainerBalancer.java | 341 ++++++++++++++++++---
.../balancer/ContainerBalancerConfiguration.java | 52 ++--
.../balancer/ContainerBalancerMetrics.java | 127 ++++++++
.../org/apache/hadoop/hdds/scm/ha/SCMContext.java | 13 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 1 +
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 29 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 29 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 17 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 4 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 10 +
.../hdds/scm/server/StorageContainerManager.java | 52 +++-
.../main/resources/webapps/scm/scm-overview.html | 8 +
.../hadoop/hdds/scm/container/MockNodeManager.java | 88 ++++--
.../container/balancer/TestContainerBalancer.java | 219 +++++++++++++
.../apache/hadoop/hdds/scm/ha/TestSCMContext.java | 6 +-
.../hadoop/hdds/scm/ha/TestSCMServiceManager.java | 4 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 4 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 25 ++
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 2 +-
hadoop-hdds/test-utils/pom.xml | 6 +-
hadoop-hdds/tools/pom.xml | 26 +-
.../hdds/scm/cli/ContainerOperationClient.java | 42 ++-
.../org/apache/hadoop/hdds/scm/cli/ScmOption.java | 14 +-
hadoop-ozone/client/pom.xml | 14 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 106 ++++---
.../hadoop/ozone/client/io/KeyInputStream.java | 51 ++-
.../hadoop/ozone/client/io/OzoneInputStream.java | 15 +-
.../ozone/client/protocol/ClientProtocol.java | 58 ++++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 69 +++--
hadoop-ozone/common/pom.xml | 26 +-
.../hadoop/ozone/client/io/LengthInputStream.java | 17 +-
hadoop-ozone/csi/pom.xml | 14 +-
hadoop-ozone/datanode/pom.xml | 14 +-
hadoop-ozone/dev-support/checks/coverage.sh | 3 +-
hadoop-ozone/dev-support/checks/integration.sh | 2 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
hadoop-ozone/dev-support/docker/Dockerfile | 2 +-
.../intellij/runConfigurations/CsiServer.xml | 4 +-
.../intellij/runConfigurations/Datanode1.xml | 2 +-
.../intellij/runConfigurations/Datanode2.xml | 2 +-
.../intellij/runConfigurations/Datanode3.xml | 2 +-
.../intellij/runConfigurations/FreonStandalone.xml | 4 +-
.../intellij/runConfigurations/OzoneFsShell.xml | 2 +-
.../intellij/runConfigurations/OzoneManager.xml | 2 +-
.../runConfigurations/OzoneManagerInit.xml | 2 +-
.../intellij/runConfigurations/OzoneShell.xml | 2 +-
.../intellij/runConfigurations/Recon.xml | 4 +-
.../intellij/runConfigurations/S3Gateway.xml | 4 +-
.../runConfigurations/StorageContainerManager.xml | 2 +-
.../StorageContainerManagerInit.xml | 2 +-
.../dist/dev-support/bin/dist-layout-stitching | 3 +
.../dist/dev-support/bin/dist-tar-stitching | 6 +-
hadoop-ozone/dist/pom.xml | 73 ++---
.../docker-image/docker-krb5/Dockerfile-krb5 | 35 ---
.../common/docker-image/docker-krb5/README.md | 34 --
.../common/docker-image/docker-krb5/kadm5.acl | 20 --
.../common/docker-image/docker-krb5/launcher.sh | 25 --
.../dist/src/main/compose/ozone-ha/docker-config | 1 +
.../main/compose/ozone-mr/hadoop27/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop31/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop32/docker-config | 4 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 44 ++-
.../src/main/compose/ozonesecure-ha/docker-config | 24 +-
.../compose/ozonesecure-ha/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-ha/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-ha/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-ha/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-ha/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-ha/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-ha/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-ha/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-ha}/krb5.conf | 4 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 1 +
.../dist/src/main/compose/ozonesecure-mr/README.md | 2 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 30 +-
.../src/main/compose/ozonesecure-mr/docker-config | 25 +-
.../compose/ozonesecure-mr/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-mr/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-mr/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-mr/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-mr/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-mr/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-mr/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-mr/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-mr}/krb5.conf | 6 +-
.../dist/src/main/compose/ozonesecure/.env | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 23 +-
.../src/main/compose/ozonesecure/docker-config | 24 +-
.../main/compose/ozonesecure/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../src/main/compose/ozonesecure/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../src/main/compose/ozonesecure/keytabs/om.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../compose/ozonesecure/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure}/krb5.conf | 4 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 4 +-
.../definitions/ozone/definitions/persistence.yaml | 4 +-
.../examples/ozone-ha/datanode-statefulset.yaml | 2 +-
.../k8s/examples/ozone/datanode-statefulset.yaml | 2 +-
hadoop-ozone/dist/src/main/keytabs/HTTP.keytab | Bin 0 -> 432 bytes
hadoop-ozone/dist/src/main/keytabs/dn.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/hadoop.keytab | Bin 0 -> 146 bytes
hadoop-ozone/dist/src/main/keytabs/jhs.keytab | Bin 0 -> 142 bytes
hadoop-ozone/dist/src/main/keytabs/nm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/om.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/recon.keytab | Bin 0 -> 296 bytes
hadoop-ozone/dist/src/main/keytabs/rm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/s3g.keytab | Bin 0 -> 434 bytes
hadoop-ozone/dist/src/main/keytabs/scm.keytab | Bin 0 -> 586 bytes
hadoop-ozone/dist/src/main/keytabs/testuser.keytab | Bin 0 -> 450 bytes
.../dist/src/main/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../dist/src/main/keytabs/update-keytabs.sh | 77 +++++
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 2 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 58 ++--
.../dist/src/main/license/update-jar-report.sh | 2 +-
.../src/main/smoketest/admincli/container.robot | 11 +-
.../dist/src/main/smoketest/basic/links.robot | 10 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 6 +-
.../dist/src/main/smoketest/cli/classpath.robot | 14 +-
.../dist/src/main/smoketest/commonlib.robot | 8 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 10 +-
.../main/smoketest/security/ozone-secure-fs.robot | 18 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 42 +--
.../fault-injection-test/mini-chaos-tests/pom.xml | 24 +-
.../fault-injection-test/network-tests/pom.xml | 6 +-
hadoop-ozone/fault-injection-test/pom.xml | 6 +-
hadoop-ozone/insight/pom.xml | 42 +--
.../hadoop/ozone/insight/BaseInsightPoint.java | 4 +-
hadoop-ozone/integration-test/pom.xml | 70 ++---
.../ozone/client/rpc/read/TestKeyInputStream.java | 63 +++-
.../container/metrics/TestContainerMetrics.java | 2 +-
.../server/TestSecureContainerServer.java | 2 +-
.../apache/hadoop/ozone/dn/DatanodeTestUtils.java | 31 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 6 +
.../TestDatanodeHddsVolumeFailureToleration.java | 131 ++++++++
.../hadoop/ozone/shell/TestOzoneShellHA.java | 99 ++++++
hadoop-ozone/interface-client/pom.xml | 10 +-
hadoop-ozone/interface-storage/pom.xml | 30 +-
hadoop-ozone/native-client/README.md | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 60 ++--
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 24 ++
.../S3MultipartUploadCompleteRequest.java | 3 +-
hadoop-ozone/ozonefs-common/pom.xml | 42 +--
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 32 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 29 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 30 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 16 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 14 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 10 +-
hadoop-ozone/ozonefs/pom.xml | 14 +-
hadoop-ozone/pom.xml | 142 ++++-----
hadoop-ozone/recon-codegen/pom.xml | 10 +-
hadoop-ozone/recon/pom.xml | 38 +--
hadoop-ozone/s3gateway/pom.xml | 42 +--
.../hadoop/ozone/client/OzoneBucketStub.java | 5 +-
hadoop-ozone/tools/pom.xml | 50 +--
.../ozone/debug/container/ContainerCommands.java | 2 +-
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 2 +-
.../ozone/freon/ClosedContainerReplicator.java | 3 +-
.../containergenerator/GeneratorDatanode.java | 2 +-
.../ozone/genesis/BenchMarkDatanodeDispatcher.java | 2 +-
.../hadoop/ozone/shell/keys/CopyKeyHandler.java | 28 +-
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 27 +-
.../hadoop/ozone/audit/parser/TestAuditParser.java | 31 +-
.../tools/src/test/resources/testaudit.log | 3 +
.../resources/{testaudit.log => testloadaudit.log} | 24 +-
pom.xml | 4 +-
240 files changed, 3485 insertions(+), 1481 deletions(-)
delete mode 100644 .github/buildenv/Dockerfile
delete mode 100755 .github/buildenv/entrypoint.sh
create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteArrayReader.java
create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteBufferReader.java
create mode 100644 hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteReaderStrategy.java
create mode 100644 hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/client/TestReplicationConfig.java
create mode 100644 hadoop-hdds/docs/content/feature/scm-secure-ha.png
copy hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/{ContainerTokenSecretManager.java => ContainerTokenGenerator.java} (50%)
delete mode 100644 hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocol/TestReplicationConfig.java
create mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java
create mode 100644 hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancer.java
delete mode 100644 hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/Dockerfile-krb5
delete mode 100644 hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/README.md
delete mode 100644 hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/kadm5.acl
delete mode 100644 hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/launcher.sh
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/HTTP.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/dn.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/om.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/recon.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/s3g.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/scm.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/testuser.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/testuser2.keytab
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 => ozonesecure-ha}/krb5.conf (96%)
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/HTTP.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/dn.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/om.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/recon.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/s3g.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/scm.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/testuser.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/testuser2.keytab
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 => ozonesecure-mr}/krb5.conf (91%)
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/HTTP.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/dn.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/om.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/recon.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/s3g.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/scm.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/testuser.keytab
create mode 100755 hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/testuser2.keytab
rename hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 => ozonesecure}/krb5.conf (96%)
create mode 100755 hadoop-ozone/dist/src/main/keytabs/HTTP.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/dn.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/hadoop.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/jhs.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/nm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/om.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/recon.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/rm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/s3g.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/scm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser2.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/update-keytabs.sh
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/volume/TestDatanodeHddsVolumeFailureToleration.java
copy hadoop-ozone/tools/src/test/resources/{testaudit.log => testloadaudit.log} (69%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org
[ozone] 01/03: Merge remote-tracking branch 'origin/master' into
HDDS-3816-ec
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3816-ec
in repository https://gitbox.apache.org/repos/asf/ozone.git
commit 9c4008d5404d310ddc4e7cf5be22b80156692ebf
Merge: 132911c 0b4779c
Author: Elek Márton <el...@apache.org>
AuthorDate: Tue May 25 13:30:16 2021 +0200
Merge remote-tracking branch 'origin/master' into HDDS-3816-ec
.github/buildenv/Dockerfile | 20 --
.github/buildenv/entrypoint.sh | 18 -
.github/workflows/post-commit.yml | 117 +++++--
.gitignore | 3 +
CONTRIBUTING.md | 10 +-
NOTICE.txt | 2 +-
README.md | 2 +-
.../hadoop/hdds/scm/XceiverClientFactory.java | 3 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 8 +-
hadoop-hdds/common/pom.xml | 5 +
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 3 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 95 +++--
.../hadoop/hdds/client/RatisReplicationConfig.java | 15 +
.../hadoop/hdds/client/ReplicationConfig.java | 55 +++
.../hdds/client/StandaloneReplicationConfig.java | 9 +
.../hdds/ratis/ServerNotLeaderException.java | 15 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 +-
.../hdds/scm/container/ContainerException.java | 14 +-
.../scm/container/ContainerNotFoundException.java | 4 +-
.../ContainerReplicaNotFoundException.java | 4 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 7 +-
.../hadoop/hdds/scm/ha/SCMHAConfiguration.java | 4 +-
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 58 ++-
.../scm/pipeline/PipelineNotFoundException.java | 8 +-
.../pipeline/UnknownPipelineStateException.java | 8 +-
.../protocol/StorageContainerLocationProtocol.java | 18 +
.../security/token/ContainerTokenIdentifier.java | 103 ++++++
.../security/token/OzoneBlockTokenIdentifier.java | 146 +++-----
.../security/token/ShortLivedTokenIdentifier.java | 112 ++++++
.../hadoop/hdds/security/x509/SecurityConfig.java | 16 +
.../hadoop/hdds/security/x509/crl}/CRLCodec.java | 2 +-
.../hadoop/hdds/security/x509/crl/CRLInfo.java | 1 -
.../hdds/security/x509/crl}/CRLInfoCodec.java | 3 +-
.../hdds/security/x509/crl/package-info.java | 24 ++
.../org/apache/hadoop/hdds/utils/db/Codec.java | 0
.../apache/hadoop/hdds/utils/db/package-info.java | 22 ++
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 10 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 12 +
.../common/src/main/resources/ozone-default.xml | 164 ++++++---
.../hadoop/hdds/client/TestReplicationConfig.java | 147 ++++++++
.../hdds/ratis/TestServerNotLeaderException.java | 82 +++++
.../ozone/container/ContainerTestHelper.java | 146 ++++----
hadoop-hdds/container-service/pom.xml | 12 +
.../hdds/datanode/metadata/CRLDBDefinition.java | 99 ++++++
.../hdds/datanode/metadata/DatanodeCRLStore.java | 84 +++++
.../datanode/metadata/DatanodeCRLStoreImpl.java | 127 +++++++
.../hdds/datanode/metadata/package-info.java | 22 ++
.../apache/hadoop/ozone/HddsDatanodeService.java | 19 +-
.../container/common/impl/HddsDispatcher.java | 25 +-
.../common/statemachine/DatanodeConfiguration.java | 30 ++
.../states/endpoint/HeartbeatEndpointTask.java | 2 +-
.../states/endpoint/RegisterEndpointTask.java | 2 +-
.../container/common/utils/HddsVolumeUtil.java | 11 +
.../ozone/container/common/volume/HddsVolume.java | 13 +
.../container/common/volume/MutableVolumeSet.java | 28 +-
.../container/keyvalue/KeyValueContainer.java | 6 +
.../container/keyvalue/KeyValueContainerData.java | 2 +
.../ozone/container/keyvalue/KeyValueHandler.java | 3 +-
.../container/keyvalue/helpers/BlockUtils.java | 4 +-
.../container/keyvalue/helpers/ChunkUtils.java | 42 ++-
.../keyvalue/impl/ChunkManagerDispatcher.java | 10 +-
.../keyvalue/impl/ChunkManagerFactory.java | 5 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 32 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 15 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 21 +-
.../stream/DirectoryServerDestination.java | 35 +-
.../container/stream/DirectoryServerSource.java | 57 +++
.../container/stream/DirstreamClientHandler.java | 143 ++++++++
.../container/stream/DirstreamServerHandler.java | 135 +++++++
.../ozone/container/stream/StreamingClient.java | 101 ++++++
.../container/stream/StreamingDestination.java | 31 +-
.../ozone/container/stream/StreamingServer.java | 94 +++++
.../ozone/container/stream/StreamingSource.java | 35 +-
.../ozone/container/stream/package-info.java | 22 ++
.../metadata/TestDatanodeCRLStoreImpl.java | 127 +++++++
.../hdds/datanode/metadata/package-info.java} | 31 +-
.../hadoop/ozone/TestHddsDatanodeService.java | 6 +-
.../container/common/TestBlockDeletingService.java | 47 ++-
.../TestSchemaOneBackwardsCompatibility.java | 18 +
.../impl/TestContainerDeletionChoosingPolicy.java | 31 +-
.../common/impl/TestContainerPersistence.java | 3 +-
.../statemachine/TestDatanodeConfiguration.java | 15 +
.../container/keyvalue/ChunkLayoutTestInfo.java | 4 +-
.../container/keyvalue/helpers/TestChunkUtils.java | 14 +-
.../keyvalue/impl/TestFilePerChunkStrategy.java | 3 +-
.../container/stream/TestStreamingServer.java | 127 +++++++
hadoop-hdds/docs/content/interface/ReconApi.md | 390 +++++++++------------
hadoop-hdds/docs/content/interface/ReconApi.zh.md | 356 ++++++++-----------
hadoop-hdds/docs/content/interface/S3.md | 21 ++
hadoop-hdds/framework/pom.xml | 6 +
.../hadoop/hdds/protocol/SCMSecurityProtocol.java | 11 +
.../SCMSecurityProtocolClientSideTranslatorPB.java | 14 +
.../hadoop/hdds/scm/protocolPB/OzonePBHelper.java | 82 +++++
...inerLocationProtocolClientSideTranslatorPB.java | 64 +++-
.../SCMBlockLocationFailoverProxyProvider.java | 136 +++----
.../SCMContainerLocationFailoverProxyProvider.java | 158 +++++----
.../SCMSecurityProtocolFailoverProxyProvider.java | 101 +++---
.../hadoop/hdds}/security/OzoneSecretKey.java | 4 +-
.../hadoop/hdds}/security/OzoneSecretManager.java | 2 +-
.../hdds}/security/OzoneSecurityException.java | 2 +-
.../apache/hadoop/hdds/security/package-info.java | 31 +-
.../hdds/security/token/BlockTokenVerifier.java | 168 +++------
.../security/token/CompositeTokenVerifier.java | 47 +++
.../token/ContainerTokenSecretManager.java | 48 +++
.../security/token/ContainerTokenVerifier.java | 51 +++
.../hdds/security/token/NoopTokenVerifier.java | 38 +-
.../token/ShortLivedTokenSecretManager.java | 107 ++++++
.../security/token/ShortLivedTokenVerifier.java | 135 +++++++
.../hadoop/hdds/security/token/TokenVerifier.java | 64 +++-
.../certificate/authority/CertificateServer.java | 4 +-
.../certificate/authority/DefaultCAServer.java | 3 +-
.../certificate/authority/DefaultCRLApprover.java | 2 +-
.../x509/certificate/client/CertificateClient.java | 15 +-
.../hadoop/hdds/server/http/BaseHttpServer.java | 2 +
.../hadoop/hdds/utils/DBCheckpointServlet.java | 30 +-
.../java/org/apache/hadoop/hdds/utils/HAUtils.java | 155 ++++----
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 4 +-
.../hdds/scm/protocol/TestReplicationConfig.java | 86 -----
.../security/token/TestBlockTokenVerifier.java | 75 ++++
.../security/token/TestContainerTokenVerifier.java | 75 ++++
.../token/TestOzoneBlockTokenIdentifier.java | 2 +-
.../hdds/security/token/TokenVerifierTests.java | 242 +++++++++++++
.../certificate/authority/TestDefaultCAServer.java | 6 +-
.../x509/certificate/utils/TestCRLCodec.java | 1 +
.../src/main/proto/ScmAdminProtocol.proto | 31 ++
.../interface-client/src/main/proto/hdds.proto | 16 +
.../src/main/proto/ScmServerProtocol.proto | 5 +
.../src/main/proto/ScmServerSecurityProtocol.proto | 28 ++
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 16 +-
.../java/org/apache/hadoop/hdds/scm/ScmUtils.java | 51 ++-
.../hdds/scm/container/ContainerReportHandler.java | 53 +--
.../scm/container/ContainerStateManagerImpl.java | 11 +-
.../IncrementalContainerReportHandler.java | 54 +--
.../hadoop/hdds/scm/ha/InterSCMGrpcClient.java | 30 +-
.../hdds/scm/ha/InterSCMGrpcProtocolService.java | 22 ++
.../hadoop/hdds/scm/ha/MockSCMHAManager.java | 16 +-
.../apache/hadoop/hdds/scm/ha/SCMHAManager.java | 29 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 192 +++++-----
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 7 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 35 +-
.../hadoop/hdds/scm/ha/SCMSnapshotDownloader.java | 5 +-
.../hadoop/hdds/scm/ha/SCMSnapshotProvider.java | 19 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 94 +++--
.../hadoop/hdds/scm/metadata/SCMDBDefinition.java | 1 +
.../hdds/scm/node/NodeDecommissionManager.java | 31 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 16 +
...lockLocationProtocolServerSideTranslatorPB.java | 2 +-
...inerLocationProtocolServerSideTranslatorPB.java | 49 ++-
.../hdds/scm/server/SCMClientProtocolServer.java | 92 +++--
.../hdds/scm/server/SCMDBCheckpointServlet.java | 5 +-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 33 +-
.../hdds/scm/server/StorageContainerManager.java | 143 ++++++--
.../hadoop/hdds/scm/container/MockNodeManager.java | 21 +-
.../scm/container/TestContainerReportHandler.java | 4 +-
.../TestIncrementalContainerReportHandler.java | 96 ++++-
.../algorithms/TestContainerPlacementFactory.java | 51 ++-
.../TestSCMContainerPlacementCapacity.java | 37 +-
.../TestSCMContainerPlacementRackAware.java | 105 ++++--
.../TestSCMContainerPlacementRandom.java | 32 +-
.../hadoop/hdds/scm/ha/TestSCMHAConfiguration.java | 76 ++++
.../hdds/scm/node/TestNodeDecommissionManager.java | 33 ++
.../scm/pipeline/TestRatisPipelineProvider.java | 4 +-
.../hdds/scm/cli/ContainerOperationClient.java | 30 +-
.../apache/hadoop/ozone/client/ObjectStore.java | 4 +
.../apache/hadoop/ozone/client/OzoneBucket.java | 121 ++++---
.../hadoop/ozone/client/OzoneKeyLocation.java | 15 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 57 ++-
.../client/io/BlockOutputStreamEntryPool.java | 2 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 5 +-
.../ozone/client/protocol/ClientProtocol.java | 65 ++++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 119 +++++--
.../hadoop/ozone/client/MockBlockAllocator.java} | 36 +-
.../hadoop/ozone/client/MockDatanodeStorage.java | 73 ++++
.../hadoop/ozone/client/MockOmTransport.java | 276 +++++++++++++++
.../ozone/client/MockXceiverClientFactory.java | 72 ++++
.../hadoop/ozone/client/MockXceiverClientSpi.java | 180 ++++++++++
.../ozone/client/SinglePipelineBlockAllocator.java | 90 +++++
.../hadoop/ozone/client/TestOzoneClient.java | 174 +++++++++
.../main/java/org/apache/hadoop/ozone/OFSPath.java | 3 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 1 +
.../org/apache/hadoop/ozone/audit/OMAction.java | 3 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 39 +--
.../hadoop/ozone/om/helpers/OzoneAclUtil.java | 20 ++
.../ozone/om/protocol/OzoneManagerProtocol.java | 8 +
...OzoneManagerProtocolClientSideTranslatorPB.java | 11 +
.../ozone/security/acl/IAccessAuthorizer.java | 1 -
hadoop-ozone/dev-support/checks/_lib.sh | 2 +-
hadoop-ozone/dev-support/checks/build.sh | 2 +-
hadoop-ozone/dev-support/checks/dependency.sh | 50 +++
hadoop-ozone/dev-support/checks/junit.sh | 2 +-
.../src/main/compose/ozone-ha/docker-compose.yaml | 10 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 16 +-
.../src/main/compose/ozonesecure-ha/docker-config | 3 +-
.../dist/src/main/compose/ozonesecure-ha/test.sh | 5 +
.../src/main/compose/ozonesecure-mr/docker-config | 1 +
.../src/main/compose/ozonesecure/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 4 +-
hadoop-ozone/dist/src/main/license/bin/NOTICE.txt | 2 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 235 +++++++++++++
.../test.sh => license/update-jar-report.sh} | 28 +-
.../src/main/smoketest/admincli/container.robot | 14 +-
.../src/main/smoketest/admincli/datanode.robot | 6 +-
.../src/main/smoketest/admincli/pipeline.robot | 7 +-
.../smoketest/admincli/replicationmanager.robot | 7 +-
.../src/main/smoketest/admincli/safemode.robot | 6 +-
.../dist/src/main/smoketest/admincli/scmha.robot | 2 +-
.../dist/src/main/smoketest/cli/envvars.robot | 15 +
.../dist/src/main/smoketest/commonlib.robot | 11 +
.../dist/src/main/smoketest/freon/remove.robot | 43 +++
hadoop-ozone/dist/src/shell/ozone/ozone | 7 +-
.../dist/src/shell/ozone/ozone-functions.sh | 3 +-
.../hdds/scm/TestSCMDbCheckpointServlet.java | 9 +-
.../hadoop/hdds/scm/TestSCMInstallSnapshot.java | 12 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 2 +-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 33 ++
.../hadoop/ozone/TestContainerOperations.java | 21 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 5 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 78 ++++-
.../hadoop/ozone/TestStorageContainerManager.java | 32 +-
.../ozone/client/rpc/Test2WayCommitInRatis.java | 3 +-
.../TestContainerStateMachineFailureOnRead.java | 6 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 10 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 24 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 12 -
.../ozone/client/rpc/TestWatchForCommit.java | 3 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 11 +-
.../ozoneimpl/TestSecureOzoneContainer.java | 151 ++++----
.../server/TestSecureContainerServer.java | 234 ++++++++-----
.../apache/hadoop/ozone/dn/DatanodeTestUtils.java | 243 +++++++++++++
.../TestDatanodeHddsVolumeFailureDetection.java | 289 +++++++++++++++
.../ozone/freon/TestFreonWithDatanodeRestart.java | 126 -------
.../ozone/freon/TestHadoopDirTreeGenerator.java | 2 +-
.../ozone/freon/TestHadoopNestedDirGenerator.java | 5 +-
.../ozone/freon/TestOzoneClientKeyGenerator.java | 94 -----
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 19 +
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 274 +++++++++------
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 2 +-
.../hadoop/ozone/scm/TestFailoverWithSCMHA.java | 2 +-
.../ozone/scm/TestSCMInstallSnapshotWithHA.java | 73 ++--
.../src/main/proto/OmClientProtocol.proto | 8 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 8 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 27 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 95 +++--
.../hadoop/ozone/om/S3SecretManagerImpl.java | 4 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 3 +
.../om/request/bucket/OMBucketDeleteRequest.java | 2 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 9 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 2 +-
.../S3MultipartUploadCommitPartRequest.java | 2 +-
.../om/request/s3/security/S3GetSecretRequest.java | 14 +-
.../request/s3/security/S3RevokeSecretRequest.java | 155 ++++++++
.../s3/security/S3RevokeSecretResponse.java | 57 +++
...OzoneManagerProtocolServerSideTranslatorPB.java | 27 +-
.../security/OzoneBlockTokenSecretManager.java | 86 +----
.../OzoneDelegationTokenSecretManager.java | 1 +
.../security/TestOzoneBlockTokenSecretManager.java | 285 ++++++++-------
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 32 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 32 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 10 -
hadoop-ozone/ozonefs-hadoop3/pom.xml | 10 -
hadoop-ozone/ozonefs/pom.xml | 10 -
hadoop-ozone/recon/pom.xml | 2 +-
.../ozone/recon/scm/ReconContainerManager.java | 107 ++++--
.../recon/scm/ReconContainerReportHandler.java | 24 +-
.../ReconIncrementalContainerReportHandler.java | 2 +-
.../recon/spi/StorageContainerServiceProvider.java | 8 +
.../impl/StorageContainerServiceProviderImpl.java | 6 +
.../ozone/recon/api/TestContainerEndpoint.java | 2 +-
.../hadoop/ozone/recon/api/TestEndpoints.java | 8 +
.../scm/AbstractReconContainerManagerTest.java | 30 ++
.../ozone/recon/scm/TestReconContainerManager.java | 50 ++-
...TestReconIncrementalContainerReportHandler.java | 3 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 17 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 283 +++++++++++++--
.../s3/endpoint/ListMultipartUploadsResult.java | 53 +--
.../endpoint/PutBucketAclRequestUnmarshaller.java | 85 +++++
.../org/apache/hadoop/ozone/s3/endpoint/S3Acl.java | 342 ++++++++++++++++++
.../hadoop/ozone/s3/endpoint/S3BucketAcl.java | 235 +++++++++++++
.../apache/hadoop/ozone/s3/endpoint/S3Owner.java | 72 ++++
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 5 +
.../hadoop/ozone/client/OzoneBucketStub.java | 28 +-
.../hadoop/ozone/client/OzoneVolumeStub.java | 24 ++
.../hadoop/ozone/s3/TestOzoneClientProducer.java | 4 +-
.../hadoop/ozone/s3/endpoint/TestBucketAcl.java | 266 ++++++++++++++
.../{TestBucketGet.java => TestBucketList.java} | 76 ++--
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 6 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 56 ++-
.../src/test/resources/groupAccessControlList.xml | 57 +++
.../src/test/resources/userAccessControlList.xml | 42 +++
.../org/apache/hadoop/ozone/debug/OzoneDebug.java | 10 +
.../ContainerCommands.java} | 130 +++----
.../ozone/debug/container/ExportSubcommand.java | 75 ++++
.../ozone/debug/container/InfoSubcommand.java | 53 +++
.../container/ListSubcommand.java} | 53 ++-
.../hadoop/ozone/debug/container/package-info.java | 22 ++
.../hadoop/ozone/freon/BaseFreonGenerator.java | 7 +
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 2 +-
.../java/org/apache/hadoop/ozone/freon/Freon.java | 5 +-
.../hadoop/ozone/freon/OmBucketGenerator.java | 2 +-
...OmBucketGenerator.java => OmBucketRemover.java} | 36 +-
...etGenerator.java => OzoneClientKeyRemover.java} | 60 ++--
.../hadoop/ozone/freon/RandomKeyGenerator.java | 137 ++++++++
.../apache/hadoop/ozone/freon/S3KeyGenerator.java | 1 +
.../hadoop/ozone/freon/StreamingGenerator.java | 144 ++++++++
.../containergenerator/GeneratorDatanode.java | 2 +-
.../GenerateOzoneRequiredConfigurations.java | 44 ++-
.../ozone/genesis/BenchmarkChunkManager.java | 4 +-
.../hadoop/ozone/shell/keys/CopyKeyHandler.java | 28 +-
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 27 +-
.../hadoop/ozone/shell/s3/GetS3SecretHandler.java | 11 +-
...cretHandler.java => RevokeS3SecretHandler.java} | 57 +--
.../org/apache/hadoop/ozone/shell/s3/S3Shell.java | 3 +-
.../TestGenerateOzoneRequiredConfigurations.java | 52 ++-
pom.xml | 59 +---
315 files changed, 11476 insertions(+), 3462 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org
[ozone] 03/03: Merge remote-tracking branch 'origin/master' into
HDDS-3816-ec
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3816-ec
in repository https://gitbox.apache.org/repos/asf/ozone.git
commit f904539c0fa0798f0fbc4c09e5ae98bf06d1a91e
Merge: c960ec7 69b92a0
Author: Elek Márton <el...@apache.org>
AuthorDate: Wed Jun 2 11:35:59 2021 +0200
Merge remote-tracking branch 'origin/master' into HDDS-3816-ec
.github/workflows/post-commit.yml | 26 +-
hadoop-hdds/client/pom.xml | 30 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 29 +-
.../hadoop/hdds/scm/storage/ByteArrayReader.java | 67 ++++
.../hadoop/hdds/scm/storage/ByteBufferReader.java | 71 +++++
.../hdds/scm/storage/ByteReaderStrategy.java | 41 +++
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 38 ++-
.../hdds/scm/storage/TestBlockInputStream.java | 35 +++
hadoop-hdds/common/pom.xml | 26 +-
.../apache/hadoop/hdds/DFSConfigKeysLegacy.java | 5 -
.../java/org/apache/hadoop/hdds/HddsUtils.java | 17 +
hadoop-hdds/config/pom.xml | 10 +-
hadoop-hdds/container-service/pom.xml | 42 +--
.../container/common/report/ReportPublisher.java | 4 +-
.../common/statemachine/DatanodeConfiguration.java | 30 ++
.../common/statemachine/DatanodeStateMachine.java | 6 +
.../CloseContainerCommandHandler.java | 9 +-
.../states/endpoint/HeartbeatEndpointTask.java | 4 +
.../container/common/volume/HddsVolumeChecker.java | 11 +-
.../container/common/volume/MutableVolumeSet.java | 42 ++-
.../container/keyvalue/helpers/ChunkUtils.java | 12 +
.../keyvalue/impl/ChunkManagerDummyImpl.java | 6 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 5 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 2 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 5 +-
.../hadoop/ozone/protocol/commands/SCMCommand.java | 10 +
.../container/common/TestBlockDeletingService.java | 2 +-
.../TestSchemaOneBackwardsCompatibility.java | 4 +-
.../common/impl/TestContainerPersistence.java | 2 +-
.../container/common/impl/TestHddsDispatcher.java | 5 +-
.../common/report/TestReportPublisher.java | 9 +-
.../statemachine/TestDatanodeConfiguration.java | 12 +
.../container/common/volume/TestVolumeSet.java | 5 +-
.../common/volume/TestVolumeSetDiskChecks.java | 11 +-
.../keyvalue/TestKeyValueBlockIterator.java | 2 +-
.../keyvalue/TestKeyValueContainerCheck.java | 2 +-
.../container/keyvalue/TestKeyValueHandler.java | 3 +-
.../keyvalue/impl/CommonChunkManagerTestCases.java | 33 ++
.../container/ozoneimpl/TestContainerReader.java | 2 +-
.../container/ozoneimpl/TestOzoneContainer.java | 3 +-
hadoop-hdds/docs/content/feature/SCM-HA.md | 68 +++-
hadoop-hdds/docs/content/feature/scm-secure-ha.png | Bin 0 -> 46757 bytes
hadoop-hdds/docs/content/interface/O3fs.md | 4 +-
hadoop-hdds/docs/content/interface/O3fs.zh.md | 8 +-
hadoop-hdds/docs/content/interface/Ofs.md | 4 +-
hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md | 6 +-
.../docs/content/recipe/SparkOzoneFSK8S.zh.md | 6 +-
hadoop-hdds/docs/pom.xml | 6 +-
hadoop-hdds/framework/pom.xml | 34 +-
...etManager.java => ContainerTokenGenerator.java} | 58 ++--
.../token/ContainerTokenSecretManager.java | 30 +-
hadoop-hdds/hadoop-dependency-client/pom.xml | 6 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 6 +-
hadoop-hdds/hadoop-dependency-test/pom.xml | 6 +-
hadoop-hdds/interface-admin/pom.xml | 10 +-
hadoop-hdds/interface-client/pom.xml | 6 +-
hadoop-hdds/interface-server/pom.xml | 10 +-
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 1 +
hadoop-hdds/pom.xml | 78 ++---
hadoop-hdds/server-scm/pom.xml | 56 ++--
.../scm/container/CloseContainerEventHandler.java | 9 +
.../hdds/scm/container/ReplicationManager.java | 12 +-
.../scm/container/balancer/ContainerBalancer.java | 341 ++++++++++++++++++---
.../balancer/ContainerBalancerConfiguration.java | 52 ++--
.../balancer/ContainerBalancerMetrics.java | 127 ++++++++
.../org/apache/hadoop/hdds/scm/ha/SCMContext.java | 13 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 1 +
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 29 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 29 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 17 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 4 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 10 +
.../hdds/scm/server/StorageContainerManager.java | 52 +++-
.../main/resources/webapps/scm/scm-overview.html | 8 +
.../hadoop/hdds/scm/container/MockNodeManager.java | 88 ++++--
.../container/balancer/TestContainerBalancer.java | 219 +++++++++++++
.../apache/hadoop/hdds/scm/ha/TestSCMContext.java | 6 +-
.../hadoop/hdds/scm/ha/TestSCMServiceManager.java | 4 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 4 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 25 ++
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 2 +-
hadoop-hdds/test-utils/pom.xml | 6 +-
hadoop-hdds/tools/pom.xml | 26 +-
.../hdds/scm/cli/ContainerOperationClient.java | 42 ++-
.../org/apache/hadoop/hdds/scm/cli/ScmOption.java | 14 +-
hadoop-ozone/client/pom.xml | 14 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 51 ++-
.../hadoop/ozone/client/io/OzoneInputStream.java | 15 +-
hadoop-ozone/common/pom.xml | 26 +-
.../hadoop/ozone/client/io/LengthInputStream.java | 17 +-
hadoop-ozone/csi/pom.xml | 14 +-
hadoop-ozone/datanode/pom.xml | 14 +-
hadoop-ozone/dev-support/checks/coverage.sh | 3 +-
hadoop-ozone/dev-support/checks/integration.sh | 2 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
hadoop-ozone/dev-support/docker/Dockerfile | 2 +-
.../intellij/runConfigurations/CsiServer.xml | 4 +-
.../intellij/runConfigurations/Datanode1.xml | 2 +-
.../intellij/runConfigurations/Datanode2.xml | 2 +-
.../intellij/runConfigurations/Datanode3.xml | 2 +-
.../intellij/runConfigurations/FreonStandalone.xml | 4 +-
.../intellij/runConfigurations/OzoneFsShell.xml | 2 +-
.../intellij/runConfigurations/OzoneManager.xml | 2 +-
.../runConfigurations/OzoneManagerInit.xml | 2 +-
.../intellij/runConfigurations/OzoneShell.xml | 2 +-
.../intellij/runConfigurations/Recon.xml | 4 +-
.../intellij/runConfigurations/S3Gateway.xml | 4 +-
.../runConfigurations/StorageContainerManager.xml | 2 +-
.../StorageContainerManagerInit.xml | 2 +-
.../dist/dev-support/bin/dist-layout-stitching | 3 +
.../dist/dev-support/bin/dist-tar-stitching | 6 +-
hadoop-ozone/dist/pom.xml | 73 ++---
.../docker-image/docker-krb5/Dockerfile-krb5 | 35 ---
.../common/docker-image/docker-krb5/README.md | 34 --
.../common/docker-image/docker-krb5/kadm5.acl | 20 --
.../common/docker-image/docker-krb5/launcher.sh | 25 --
.../dist/src/main/compose/ozone-ha/docker-config | 1 +
.../main/compose/ozone-mr/hadoop27/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop31/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop32/docker-config | 4 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 44 ++-
.../src/main/compose/ozonesecure-ha/docker-config | 24 +-
.../compose/ozonesecure-ha/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-ha/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-ha/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-ha/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-ha/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-ha/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-ha/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-ha/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-ha}/krb5.conf | 4 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 1 +
.../dist/src/main/compose/ozonesecure-mr/README.md | 2 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 30 +-
.../src/main/compose/ozonesecure-mr/docker-config | 25 +-
.../compose/ozonesecure-mr/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-mr/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-mr/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-mr/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-mr/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-mr/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-mr/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-mr/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-mr}/krb5.conf | 6 +-
.../dist/src/main/compose/ozonesecure/.env | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 23 +-
.../src/main/compose/ozonesecure/docker-config | 24 +-
.../main/compose/ozonesecure/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../src/main/compose/ozonesecure/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../src/main/compose/ozonesecure/keytabs/om.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../compose/ozonesecure/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure}/krb5.conf | 4 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 4 +-
.../definitions/ozone/definitions/persistence.yaml | 4 +-
.../examples/ozone-ha/datanode-statefulset.yaml | 2 +-
.../k8s/examples/ozone/datanode-statefulset.yaml | 2 +-
hadoop-ozone/dist/src/main/keytabs/HTTP.keytab | Bin 0 -> 432 bytes
hadoop-ozone/dist/src/main/keytabs/dn.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/hadoop.keytab | Bin 0 -> 146 bytes
hadoop-ozone/dist/src/main/keytabs/jhs.keytab | Bin 0 -> 142 bytes
hadoop-ozone/dist/src/main/keytabs/nm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/om.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/recon.keytab | Bin 0 -> 296 bytes
hadoop-ozone/dist/src/main/keytabs/rm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/s3g.keytab | Bin 0 -> 434 bytes
hadoop-ozone/dist/src/main/keytabs/scm.keytab | Bin 0 -> 586 bytes
hadoop-ozone/dist/src/main/keytabs/testuser.keytab | Bin 0 -> 450 bytes
.../dist/src/main/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../dist/src/main/keytabs/update-keytabs.sh | 77 +++++
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 2 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 58 ++--
.../dist/src/main/license/update-jar-report.sh | 2 +-
.../src/main/smoketest/admincli/container.robot | 11 +-
.../dist/src/main/smoketest/basic/links.robot | 10 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 6 +-
.../dist/src/main/smoketest/cli/classpath.robot | 14 +-
.../dist/src/main/smoketest/commonlib.robot | 8 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 10 +-
.../main/smoketest/security/ozone-secure-fs.robot | 18 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 42 +--
.../fault-injection-test/mini-chaos-tests/pom.xml | 24 +-
.../fault-injection-test/network-tests/pom.xml | 6 +-
hadoop-ozone/fault-injection-test/pom.xml | 6 +-
hadoop-ozone/insight/pom.xml | 42 +--
.../hadoop/ozone/insight/BaseInsightPoint.java | 4 +-
hadoop-ozone/integration-test/pom.xml | 70 ++---
.../ozone/client/rpc/read/TestKeyInputStream.java | 63 +++-
.../container/metrics/TestContainerMetrics.java | 2 +-
.../server/TestSecureContainerServer.java | 2 +-
.../apache/hadoop/ozone/dn/DatanodeTestUtils.java | 31 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 6 +
.../TestDatanodeHddsVolumeFailureToleration.java | 131 ++++++++
.../hadoop/ozone/shell/TestOzoneShellHA.java | 99 ++++++
hadoop-ozone/interface-client/pom.xml | 10 +-
hadoop-ozone/interface-storage/pom.xml | 30 +-
hadoop-ozone/native-client/README.md | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 60 ++--
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 24 ++
.../S3MultipartUploadCompleteRequest.java | 3 +-
hadoop-ozone/ozonefs-common/pom.xml | 42 +--
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 30 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 16 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 14 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 10 +-
hadoop-ozone/ozonefs/pom.xml | 14 +-
hadoop-ozone/pom.xml | 142 ++++-----
hadoop-ozone/recon-codegen/pom.xml | 10 +-
hadoop-ozone/recon/pom.xml | 38 +--
hadoop-ozone/s3gateway/pom.xml | 42 +--
hadoop-ozone/tools/pom.xml | 50 +--
.../ozone/debug/container/ContainerCommands.java | 2 +-
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 2 +-
.../ozone/freon/ClosedContainerReplicator.java | 3 +-
.../containergenerator/GeneratorDatanode.java | 2 +-
.../ozone/genesis/BenchMarkDatanodeDispatcher.java | 2 +-
.../hadoop/ozone/audit/parser/TestAuditParser.java | 31 +-
.../tools/src/test/resources/testaudit.log | 3 +
.../resources/{testaudit.log => testloadaudit.log} | 24 +-
pom.xml | 4 +-
224 files changed, 2973 insertions(+), 1208 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org
[ozone] 02/03: Merge remote-tracking branch 'origin/HDDS-3816-ec'
into HDDS-3816-ec
Posted by el...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
elek pushed a commit to branch HDDS-3816-ec
in repository https://gitbox.apache.org/repos/asf/ozone.git
commit c960ec7c5af8bc1a0277675978db7389439e28c3
Merge: 9c4008d ad790b6
Author: Elek Márton <el...@apache.org>
AuthorDate: Wed Jun 2 11:35:51 2021 +0200
Merge remote-tracking branch 'origin/HDDS-3816-ec' into HDDS-3816-ec
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 5 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 20 ++++++--
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 1 +
.../container/common/helpers/ContainerUtils.java | 5 +-
.../container/common/impl/ContainerDataYaml.java | 55 ++++++++++++++++------
.../container/common/impl/HddsDispatcher.java | 5 ++
.../container/keyvalue/KeyValueContainer.java | 1 +
.../container/keyvalue/KeyValueContainerData.java | 10 ++++
.../ozone/container/keyvalue/KeyValueHandler.java | 2 +
.../ozone/container/ozoneimpl/ContainerReader.java | 4 +-
.../TestSchemaOneBackwardsCompatibility.java | 3 +-
.../common/impl/TestContainerDataYaml.java | 48 +++++++++++++++++--
12 files changed, 129 insertions(+), 30 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org