You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by jh...@apache.org on 2019/08/30 00:04:06 UTC
[hadoop] branch YARN-8200 updated (a8901a1 -> 056b54c)
This is an automated email from the ASF dual-hosted git repository.
jhung pushed a change to branch YARN-8200
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
discard a8901a1 YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda)
discard 9a8b330 YARN-9409. Port resource type changes from YARN-7237 to branch-2
discard e5d0484 YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types
discard 11f0947 YARN-9271. Backport YARN-6927 for resource type support in MapReduce
discard f0b57c8 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda)
discard 8e71ce5 YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák.
discard ab3d845 YARN-7143. FileNotFound handling in ResourceUtils is inconsistent
discard 3764197 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda)
discard e1d4d38 YARN-9291. Backport YARN-7637 to branch-2
discard 1aa537c YARN-9397. Fix empty NMResourceInfo object test failures in branch-2
discard 5ab544a YARN-7223. Document GPU isolation feature. Contributed by Wangda Tan.
discard ce74dc4 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák.
discard 61b2a2f YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda)
discard 9d875f8 YARN-9289. Backport YARN-7330 for GPU in UI to branch-2
discard a8bf897 YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung
discard 862a933 YARN-9174. Backport YARN-7224 for refactoring of GpuDevice class
discard e53ec71 YARN-9280. Backport YARN-6620 to YARN-8200/branch-2 for NodeManager-side GPU isolation
discard c0753ff YARN-9180. Port YARN-7033 NM recovery of assigned resources to branch-2
discard b8c0a65 YARN-9187. Backport YARN-6852 for GPU-specific native changes to branch-2
discard 51160ce YARN-9175. Null resources check in ResourceInfo for branch-3.0
discard 3f7d9c8 YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan.
discard 52377b0 YARN-7270 addendum: Reapplied changes after YARN-3926 backports
discard 575b86a YARN-9188. Port YARN-7136 to branch-2
discard 33785f8 YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda)
discard 04a0e99 YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R.
discard a1e7be1 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda)
discard 1d5c5f9 YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda)
discard bf3fe09 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan.
discard b3d6947 YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda)
discard 3c1e31dc0 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin.
discard d86b628 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda)
discard 907144b YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda)
discard 5489073 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan.
discard 3b35cda YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton.
discard b6328bc YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G.
discard 1c06445 YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu)
discard 5bf11f5 YARN-6788. [YARN-3926] Improve performance of resource profile branch (Contributed by Sunil Govindan via Daniel Templeton)
discard b98bc4b YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
discard b5dd7a7 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
discard 2384394 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2
discard bbd3b19 YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2
discard d9c5672 YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2
discard 0dab70c YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev.
discard 7e4bfa4 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh)
discard 77c780b YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev.
discard 76d9e7c YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev.
discard ab11d68 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev.
discard 11c5b26 YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh)
discard f3d2560 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev.
discard fceaa96 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda)
discard c15354b YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda)
add 786df37 HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy.
add 97f207d YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang.
add e786147 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen.
add f083f7f YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph
add b6be5f8 YARN-9438. launchTime not written to state store for running applications
add 8e8c16d YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06).
add e5bb849 HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
add 50953c5 HADOOP-16269. ABFS: add listFileStatus with StartFrom.
add 605c812 HADOOP-16251. ABFS: add FSMainOperationsBaseTest
add 67fe539 Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
add 42816bf HADOOP-16376. ABFS: Override access() to no-op.
add 8044415 HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
add 1376f2e HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
add 6cb7462 HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
add f56972b HADOOP-16460: ABFS: fix for Sever Name Indication (SNI)
new afcf9c5 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda)
new be2afb4 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda)
new 133eb31 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev.
new 817837e YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh)
new 7d5050a YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev.
new 015a55a YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev.
new f99d603 YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev.
new 6db8c36 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh)
new c5e77e8 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev.
new 7f614f7 YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2
new 08923e3 YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2
new d5963b2 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2
new ad6f095 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
new 1dc66db YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
new d8f3886 YARN-6788. [YARN-3926] Improve performance of resource profile branch (Contributed by Sunil Govindan via Daniel Templeton)
new 59cfecd YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu)
new 06c38ee YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G.
new 2fafdc7 YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton.
new 7b1c58b YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan.
new 43030aa YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda)
new c69dd9e YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda)
new 7c37fbe YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin.
new 1f05fa0 YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda)
new 74bb2d6 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan.
new f340726 YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda)
new 8cea242 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda)
new 45b50c4 YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R.
new 26e87be YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda)
new d6673c5 YARN-9188. Port YARN-7136 to branch-2
new 72d90e7 YARN-7270 addendum: Reapplied changes after YARN-3926 backports
new 4d30c90 YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan.
new 078dfb0 YARN-9175. Null resources check in ResourceInfo for branch-3.0
new 56259bc YARN-9187. Backport YARN-6852 for GPU-specific native changes to branch-2
new 631dfc7 YARN-9180. Port YARN-7033 NM recovery of assigned resources to branch-2
new 7ec4d7c YARN-9280. Backport YARN-6620 to YARN-8200/branch-2 for NodeManager-side GPU isolation
new 16faceb YARN-9174. Backport YARN-7224 for refactoring of GpuDevice class
new 4d82318 YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung
new 7939ae1 YARN-9289. Backport YARN-7330 for GPU in UI to branch-2
new ca7d0fd YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda)
new c539427 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák.
new 8670956 YARN-7223. Document GPU isolation feature. Contributed by Wangda Tan.
new e102c3d YARN-9397. Fix empty NMResourceInfo object test failures in branch-2
new 22ba693 YARN-9291. Backport YARN-7637 to branch-2
new 7025333 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda)
new 062fbd0 YARN-7143. FileNotFound handling in ResourceUtils is inconsistent
new 1875231 YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák.
new 6f4ebe0 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda)
new 8e5ce80 YARN-9271. Backport YARN-6927 for resource type support in MapReduce
new 28fbc4e YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types
new 87ad52f YARN-9409. Port resource type changes from YARN-7237 to branch-2
new 056b54c YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a8901a1)
\
N -- N -- N refs/heads/YARN-8200 (056b54c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 51 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev-support/bin/create-release | 2 +-
.../hdfs/qjournal/client/QuorumJournalManager.java | 14 +-
.../hdfs/qjournal/client/QuorumOutputStream.java | 5 +
.../hadoop/hdfs/server/namenode/FSEditLog.java | 3 +-
.../client/TestQuorumJournalManagerUnit.java | 61 ++++++++-
.../hdfs/server/namenode/FSImageTestUtil.java | 24 +++-
hadoop-project/pom.xml | 2 +-
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 23 +++-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 126 +++++++++++++++--
.../fs/azurebfs/constants/AbfsHttpConstants.java | 9 ++
.../constants/FileSystemConfigurations.java | 2 +-
.../fs/azurebfs/oauth2/IdentityTransformer.java | 75 ++++++++--
.../fs/azurebfs/services/AbfsOutputStream.java | 17 ++-
.../org/apache/hadoop/fs/azurebfs/utils/CRC64.java | 60 ++++++++
.../fs/azurebfs/AbstractAbfsIntegrationTest.java | 7 +-
.../fs/azurebfs/ITestAbfsIdentityTransformer.java | 58 +++++++-
.../ITestAzureBlobFileSystemFileStatus.java | 18 +++
.../ITestAzureBlobFileSystemMainOperation.java | 71 ++++++++++
...zureBlobFileSystemStoreListStatusWithRange.java | 151 +++++++++++++++++++++
.../TestAbfsConfigurationFieldsValidation.java | 6 +
.../apache/hadoop/fs/azurebfs/TestAbfsCrc64.java | 26 ++--
.../yarn/server/resourcemanager/RMAppManager.java | 2 +
.../server/resourcemanager/rmapp/RMAppImpl.java | 6 +
.../resourcemanager/scheduler/QueueMetrics.java | 23 ++++
.../scheduler/capacity/CapacityScheduler.java | 2 +
.../scheduler/capacity/UsersManager.java | 9 +-
.../capacity/conf/ZKConfigurationStore.java | 2 +-
.../yarn/server/resourcemanager/TestRMRestart.java | 19 +--
.../rmapp/TestRMAppTransitions.java | 22 +++
.../TestCapacitySchedulerSurgicalPreemption.java | 5 +
.../capacity/conf/TestZKConfigurationStore.java | 2 +
31 files changed, 783 insertions(+), 69 deletions(-)
create mode 100644 hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/CRC64.java
create mode 100644 hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
create mode 100644 hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemStoreListStatusWithRange.java
copy hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TestJUnitSetup.java => hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsCrc64.java (59%)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org