You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by el...@apache.org on 2019/09/25 09:00:16 UTC
[hadoop] branch HDDS-2067 updated (e4d4fca -> 63c730a)
This is an automated email from the ASF dual-hosted git repository.
elek pushed a change to branch HDDS-2067
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
omit e4d4fca package-info fix + removing FWIO
omit 5b359af remove RpcController
omit adee693 Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java
omit 7a44cdc Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java
omit 7256c69 HDDS-2067. Create generic service facade with tracing/metrics/logging support
add aa664d7 HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE (#1302). Contributed by Gabor Bota.
add 2b5fc95 HADOOP-16591 Fix S3A ITest*MRjob failures.
add c30e495 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar.
add 3d78b12 YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar
add 3fd3d74 HDDS-2161. Create RepeatedKeyInfo structure to be saved in deletedTable
add 6cbe5d3 HDDS-2160. Add acceptance test for ozonesecure-mr compose. Contributed by Xiaoyu Yao. (#1490)
add 0a716bd HDDS-2159. Fix Race condition in ProfileServlet#pid.
add bfe1dac HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496)
add e8e7d7b HADOOP-16561. [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes (#1500)
add 8f1a135 HDDS-2081. Fix TestRatisPipelineProvider#testCreatePipelinesDnExclude. Contributed by Aravindan Vijayan. (#1506)
add 51c64b3 HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running
add 91f50b9 HDDS-2167. Hadoop31-mr acceptance test is failing due to the shading
add 43203b4 HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun.
add 816d3cb HDFS-14837. Review of Block.java. Contributed by David Mollitor.
add afa1006 HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor.
add f16cf87 HDDS-2170. Add Object IDs and Update ID to Volume Object (#1510)
add eb96a30 HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena.
add 66400c1 HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena.
add c2731d4 YARN-9730. Support forcing configured partitions to be exclusive based on app node label
add 6917754 HDDS-2172.Ozone shell should remove description about REST protocol support. Contributed by Siddharth Wagle.
add a346381 HDDS-2168. TestOzoneManagerDoubleBufferWithOMResponse sometimes fails with out of memory error (#1509)
add 3f89084 HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs.
add bec0864 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal
add c724577 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko
add 5d472d1 HDDS-2067. Create generic service facade with tracing/metrics/logging support
add e20836e Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java
add 89d5e22 Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java
add c17eb3e remove RpcController
add ed6ac73 package-info fix + removing FWIO
add 63c730a fix checkstyle issue
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e4d4fca)
\
N -- N -- N refs/heads/HDDS-2067 (63c730a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/hadoop/fs/shell/Mkdir.java | 11 +-
.../apache/hadoop/util/NodeHealthScriptRunner.java | 1 +
.../src/site/markdown/DeprecatedProperties.md | 4 +
.../hadoop/util/TestNodeHealthScriptRunner.java | 9 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../server/OzoneProtocolMessageDispatcher.java | 1 -
.../apache/hadoop/hdds/server/ProfileServlet.java | 10 +-
.../org/apache/hadoop/hdfs/protocol/Block.java | 141 ++++++++----
.../fs/http/server/HttpFSAuthenticationFilter.java | 16 +-
.../fs/http/server/HttpFSServerWebServer.java | 22 ++
.../src/main/resources/httpfs-default.xml | 20 +-
.../server/federation/router/TestRouterQuota.java | 2 +-
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 5 +-
.../apache/hadoop/hdfs/net/DFSNetworkTopology.java | 3 +
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +-
.../hadoop/hdfs/qjournal/client/QuorumCall.java | 18 ++
.../hdfs/qjournal/client/QuorumJournalManager.java | 2 +
.../hdfs/server/blockmanagement/BlockManager.java | 13 +-
.../blockmanagement/BlockReportLeaseManager.java | 8 +-
.../src/main/resources/hdfs-default.xml | 8 +
.../hadoop/hdfs/net/TestDFSNetworkTopology.java | 16 ++
.../hadoop/hdfs/qjournal/MiniJournalCluster.java | 6 +-
.../qjournal/client/TestQuorumJournalManager.java | 34 ++-
.../hadoop-mapreduce-client-common/pom.xml | 34 +--
.../src/main/proto/HSAdminRefreshProtocol.proto | 3 +-
.../src/main/proto/MRClientProtocol.proto | 1 +
.../src/main/proto/mr_protos.proto | 1 +
.../src/main/proto/mr_service_protos.proto | 1 +
.../hadoop-mapreduce-client-shuffle/pom.xml | 27 +--
.../src/main/proto/ShuffleHandlerRecovery.proto | 1 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 6 +-
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 52 +++++
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 100 +++++++--
.../hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java | 91 ++++++++
.../src/main/proto/OzoneManagerProtocol.proto | 14 +-
.../{ozone-mr/hadoop32 => ozonesecure-mr}/test.sh | 12 +-
.../smoketest/{kinit.robot => kinit-hadoop.robot} | 2 +-
hadoop-ozone/dist/src/main/smoketest/kinit.robot | 5 +-
.../dist/src/main/smoketest/mapreduce.robot | 2 +-
.../scm/pipeline/TestRatisPipelineProvider.java | 26 ++-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 47 +++-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 90 ++++----
.../ozone/om/request/key/OMKeyDeleteRequest.java | 3 +-
.../multipart/S3MultipartUploadAbortRequest.java | 5 +-
.../S3MultipartUploadCommitPartRequest.java | 4 +-
.../om/request/volume/OMVolumeCreateRequest.java | 5 +
.../ozone/om/response/key/OMKeyDeleteResponse.java | 27 ++-
.../multipart/S3MultipartUploadAbortResponse.java | 21 +-
.../S3MultipartUploadCommitPartResponse.java | 34 ++-
.../org/apache/hadoop/ozone/web/ozShell/Shell.java | 3 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 21 +-
.../ozone/om/request/TestOMRequestUtils.java | 14 +-
.../request/volume/TestOMVolumeCreateRequest.java | 10 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 20 +-
.../s3/multipart/TestS3MultipartResponse.java | 3 +-
.../TestS3MultipartUploadAbortResponse.java | 19 +-
hadoop-ozone/ozonefs/pom.xml | 4 +
.../fs/s3a/commit/AbstractYarnClusterITest.java | 5 +-
.../fs/azurebfs/ITestAzureBlobFileSystemCLI.java | 65 ++++++
.../org/apache/hadoop/tools/DistCpConstants.java | 6 +
.../apache/hadoop/tools/mapred/CopyCommitter.java | 10 +-
.../org/apache/hadoop/tools/mapred/CopyMapper.java | 3 +-
.../tools/mapred/RetriableFileCopyCommand.java | 34 ++-
.../org/apache/hadoop/tools/util/DistCpUtils.java | 26 ++-
.../hadoop/tools/mapred/TestCopyCommitter.java | 5 +-
.../apache/hadoop/tools/mapred/TestCopyMapper.java | 56 +++++
.../tools/mapred/TestRetriableFileCopyCommand.java | 25 ++-
.../apache/hadoop/tools/util/TestDistCpUtils.java | 45 ++--
.../tools/util/TestDistCpUtilsWithCombineMode.java | 4 +-
.../hadoop-yarn/hadoop-yarn-api/pom.xml | 41 +---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 6 +
.../src/main/proto/YarnCsiAdaptor.proto | 1 +
.../main/proto/application_history_client.proto | 1 +
.../main/proto/applicationclient_protocol.proto | 1 +
.../main/proto/applicationmaster_protocol.proto | 1 +
.../src/main/proto/client_SCM_protocol.proto | 1 +
.../main/proto/containermanagement_protocol.proto | 1 +
.../src/main/proto/server/SCM_Admin_protocol.proto | 1 +
.../proto/server/application_history_server.proto | 1 +
.../resourcemanager_administration_protocol.proto | 1 +
...arn_server_resourcemanager_service_protos.proto | 1 +
.../src/main/proto/yarn_csi_adaptor.proto | 4 +-
.../src/main/proto/yarn_protos.proto | 1 +
.../src/main/proto/yarn_service_protos.proto | 1 +
.../hadoop-yarn-services-core/pom.xml | 22 +-
.../src/main/proto/ClientAMProtocol.proto | 1 +
.../hadoop-yarn/hadoop-yarn-client/pom.xml | 31 +--
.../apache/hadoop/yarn/client/cli/TestLogsCLI.java | 59 ++++-
.../src/test/proto/test_amrm_token.proto | 1 +
.../hadoop-yarn/hadoop-yarn-common/pom.xml | 41 ++--
.../hadoop/yarn/logaggregation/LogToolUtils.java | 60 +++--
.../src/main/proto/yarn_security_token.proto | 1 +
.../src/main/resources/yarn-default.xml | 9 +
.../logaggregation/TestAggregatedLogFormat.java | 42 +++-
.../TestLogAggregationIndexedFileController.java | 31 ++-
.../pom.xml | 39 ++--
.../yarn_server_timelineserver_recovery.proto | 1 +
.../hadoop-yarn-server-common/pom.xml | 39 +---
.../src/main/proto/ResourceTracker.proto | 1 +
.../src/main/proto/SCMUploader.proto | 1 +
.../main/proto/collectornodemanager_protocol.proto | 1 +
.../proto/distributed_scheduling_am_protocol.proto | 2 +-
.../src/main/proto/yarn_server_common_protos.proto | 1 +
.../proto/yarn_server_common_service_protos.proto | 1 +
.../main/proto/yarn_server_federation_protos.proto | 1 +
.../hadoop-yarn-server-nodemanager/pom.xml | 34 +--
.../src/main/proto/LocalizationProtocol.proto | 1 +
.../proto/yarn_server_nodemanager_recovery.proto | 1 +
.../yarn_server_nodemanager_service_protos.proto | 1 +
.../logaggregation/TestLogAggregationService.java | 106 +++++----
.../nodemanager/webapp/TestNMWebServices.java | 43 ++--
.../hadoop-yarn-server-resourcemanager/pom.xml | 69 +++---
.../server/resourcemanager/ClientRMService.java | 6 +-
.../resourcemanager/DefaultAMSProcessor.java | 6 +
.../yarn/server/resourcemanager/RMAppManager.java | 6 +
.../yarn/server/resourcemanager/RMAuditLogger.java | 53 ++++-
.../yarn/server/resourcemanager/RMContext.java | 4 +
.../yarn/server/resourcemanager/RMContextImpl.java | 15 ++
.../scheduler/SchedulerApplicationAttempt.java | 9 +
.../resourcemanager/scheduler/SchedulerUtils.java | 23 ++
.../capacity/CapacitySchedulerConfiguration.java | 7 +
.../scheduler/capacity/LeafQueue.java | 8 +-
.../policy/AbstractComparatorOrderingPolicy.java | 8 +-
.../FifoOrderingPolicyWithExclusivePartitions.java | 144 ++++++++++++
.../IteratorSelector.java} | 32 +--
.../scheduler/policy/OrderingPolicy.java | 3 +-
.../scheduler/policy/SchedulableEntity.java | 5 +
.../yarn_server_resourcemanager_recovery.proto | 1 +
.../server/resourcemanager/TestRMAuditLogger.java | 21 +-
.../scheduler/TestSchedulerUtils.java | 142 ++++++++++++
.../scheduler/capacity/TestCapacityScheduler.java | 11 +-
.../scheduler/capacity/TestLeafQueue.java | 145 +++++++++++-
.../scheduler/policy/MockSchedulableEntity.java | 15 +-
.../scheduler/policy/TestFairOrderingPolicy.java | 12 +-
.../scheduler/policy/TestFifoOrderingPolicy.java | 2 +-
.../TestFifoOrderingPolicyForPendingApps.java | 5 +-
...tFifoOrderingPolicyWithExclusivePartitions.java | 244 +++++++++++++++++++++
.../src/test/proto/test_client_tokens.proto | 1 +
.../hadoop-yarn-server-tests/pom.xml | 31 +--
.../src/test/proto/test_token.proto | 1 +
.../src/site/markdown/NodeManager.md | 8 +-
141 files changed, 2306 insertions(+), 723 deletions(-)
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/RepeatedOmKeyInfoCodec.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java
copy hadoop-ozone/dist/src/main/compose/{ozone-mr/hadoop32 => ozonesecure-mr}/test.sh (83%)
copy hadoop-ozone/dist/src/main/smoketest/{kinit.robot => kinit-hadoop.robot} (94%)
create mode 100644 hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCLI.java
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicyWithExclusivePartitions.java
copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/{event/ReleaseContainerEvent.java => policy/IteratorSelector.java} (62%)
create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFifoOrderingPolicyWithExclusivePartitions.java
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org