You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/03/01 16:59:51 UTC
[GitHub] [hadoop] monthonk commented on pull request #3877: Hadoop 12020. Add configurable storage class option
monthonk commented on pull request #3877:
URL: https://github.com/apache/hadoop/pull/3877#issuecomment-1055652784
Thank you Danny for reviewing the PR.
From your suggestion, I have added a change to address the InitiateMultipartUpload as well as a test for it. We used MultipartUpload when writing big files, so we have to enable scale test to verify it.
I run the integration test again with
```
<property>
<name>fs.s3a.scale.test.enabled</name>
<value>true</value>
</property>
```
Got some failures from the test, but I think it comes from other PRs because I run the test with the same configuration on trunk and also got these failures. Please see details below.
```
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAuditManagerDisabled.testAuditSpansAreAllTheSame:72 [audit span 1]
Expecting:
<NoopSpan{id=''name='no-op', path1='null', path2='null'}>
and actual:
<WrappingAuditSpan{span=LoggingAuditSpan{, id='6392e717-eb50-4cd1-8a43-564c95728668-00000010'description='https://audit.example.org/hadoop/1/span1/6392e717-eb50-4cd1-8a43-564c95728668-00000010/?op=span1&pr=monthonk&ps=ab6c3e38-a257-4729-b374-1aae23745fd9&id=6392e717-eb50-4cd1-8a43-564c95728668-00000010&t0=44&fs=6392e717-eb50-4cd1-8a43-564c95728668&t1=44&ts=1646142498449'}, valid=true}>
to refer to the same object
[ERROR] ITestAuditManagerDisabled.testAuditorDisabled:58
Expecting:
<Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='a74cd0b8-02c1-4617-9662-faf6ef2c81df', headerEnabled=true, rejectOutOfSpan=false}}>
to be an instance of:
<org.apache.hadoop.fs.s3a.audit.impl.NoopAuditManagerS3A>
but was instance of:
<org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A>
[INFO]
[ERROR] Tests run: 1066, Failures: 2, Errors: 0, Skipped: 145
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestS3AHugeMagicCommits.test_030_postCreationAssertions:128->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Pending uploads: Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID EkU..dQ6Swy.j8cgY_2RG7yP0I95KQYFAQjFVGbhethf4KCn0g3gYaVPacd1rmrmJ4TYlRPUSsbqEhL536FD1yan7MLuU72W4OD2oRVNOYlXG9rwG4wVFuXqGX._DdFI; initiated 2022-02-25 10:22:40
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID JnWoMSZF1kqmKUegZa5IwtgVi9eASHw_WDyUY9HnPlZVxYST.5zQvqbieA9.PMTzbRIPsdq1DmcBfVSXjZovJwI4QKPsh0iwauQ4awMWoeMx8b5GHAmOdM.b43Oqnmma; initiated 2022-02-25 10:29:29
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID U7tCwMSdzg5nINM.7ZIwYLnJ9b.b0a_kaNX5wbyVs_H6jN__onT2_SwsE_JxFGroLhbZW6X27cdbdub8XzYn2i5U0UmMjV.fteFeuRKX5xelP.FkBF6A799HNVNwbjwi; initiated 2022-02-25 10:36:59
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID HgfJepgnf7woHlUG_ZvRrt8Cr8AzNRNt5OujL9QsVfWJVxaydldjV56LxO36bM30V2Fg23fGM_x5GS7qIZQnu03UZ1EJiRdiLJUpFS21PJxy3Y_z.rMMTrxismrFSLKF; initiated 2022-02-25 10:43:14
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID odTpJJDJ7nbKzGafXMY0OvLHnE_SV4NtOUeQSBydcOwaelqwHcKZL.D2g9zVQH9F_iZ16O79oVUkRck6W7tBvYfOoEJfyKnY8_TJV7RgHPiq1PrIZClkhVxbXvYSCnqO; initiated 2022-02-25 10:46:08
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID AqBwYFGO2iIkpQLNaHBW1pyhSEkpZ1Uju5e5vqg2AKSdBLKT9RJ5v0EqO4jh5a_orWnWmUCPxRH3tWTKKoXbsWmWD3S4c8jLdzc_DniiGmldRS4Ms_pRFMpg75RP4pYU; initiated 2022-02-25 10:48:21
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID JqUPTJj_KTqe6CISt4tWC5LPUTfi8Qf2srAHeQLin.FNjBVNV8PV9lSdn89DJClYxgkUxixLBtBHX.uxBJMrwpHmhxUH7pda5pIakuElGVGHGDU_ue1XTOskptUzUuut; initiated 2022-02-28 12:27:35
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID N7LzVCjdC4f._mDalNvNplCTOj6iTHMtuUGrAZzrQc8czrZKktftKSYN8eXMvxAFjpVjbWLgUq5gifVlyXHRtH8cfPzfdrx2bdb70oF4SGQaOqgTrDxN61Wo9NNvGLVW; initiated 2022-02-28 13:28:19
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID PHhnOtx_Z7qp0ipPve1Eb2QYl7VTacwVx1yqKlOH52ZqjldnAgprGhlvQ6o.dRvuqKJnybjQoW8GBuhWjIpSd.6cHconMICADmKrCLtklDmrLjsn1YJvit8vtyuIun4U; initiated 2022-02-28 13:39:50
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID brf6AAghjNph5O02ERiOAnNIugjswaehfgOkINdapWTrKCZw6KsKnMZ_pxydp9cipUq5bLRW_KiUsSg6RVTBK7j3y7HzP4k4Qs6za1rCQIE2xM8agopTuP7qe_HZs8ev; initiated 2022-02-28 13:43:50
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID H_4vSJHfCejXnhFcqEqrXGp2aN2SAbWu57qkegVzOMfJx3DMt7K2gKR2NdsqbW2iTxvBmPwBlP8r1SKihslH7f_f8R2AX_3WX1gboKhGvIkOd3fptVq1w5OgsG7RaL5Z; initiated 2022-02-28 13:46:34
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID .4OXQcbfugkrQ31rCdzc78GO8Fa18Qu6LhXRfww73UIA_mQbQC7_NJP9sgxmT7TkT2w.EhHSvSHbrpauXHPCH0TtSjt2ZyzCgvSLiyB9EJZwbXOxBEV2Ot1QlItzDgUX; initiated 2022-02-28 13:50:44
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID hQicBnFHFt6p6TOEZX6ic8TU4C65L_UVJkvozEOCrCZCpWKaasClxu3Vn8UectilLy.gQbOUV6ItaStTJxF6PrultawRiiHVwN1vbLF7jq4_aOFHHQsw6OZG_LVVkoz6; initiated 2022-02-28 15:53:08
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID guUpTMey6fA0eIP69gt5Ujo427uzyOLi5BSDPtsntjrlI0Yt98YtCkMStXRGf0LjKtHK1fULLMuilphs0ox7O5UCyyYkWrZEe2BQgRWSgvqYhqbY8raszb9JfWA0x9BA; initiated 2022-02-28 16:43:17
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID UEZenRFm0kaplpQ2lznbUnPQFELPX4opLRJbvPTTL4Eh4KzmUsDfSLUQ6NFW9eze05liDXKwKwbWk0oWasHq9cC.0F4gyai5zRsW8roEEWDvJ4BQZzA0idQKmJ.v0OD_; initiated 2022-02-28 17:25:22
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID mLLdMfXUjtSQ6xaBhI.JMoCZQg1vJijizA4zXY9UAqVStaFYNgAkudNj5nXq.n6p1sYBp9gErvfiiXz4xJmnhY_.P8rWLOKtt_9HVvMthNms17pQeKKyv.xNknaEntyA; initiated 2022-02-28 17:44:30
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID Dkn4KPh5vK0cDbE3oQhUM_eofD_RUsGWCKM5JxldMBGb0SSpBfOtrDHcstg2TjqVSTKcPeC8cJHdLCqE9QpuaTqOaOov5_uac6zXUFq_Fko_v7rbfgYgbwycdcJKWCcL; initiated 2022-02-28 18:27:19
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID kpVZgERtnVh1R_0wnVS4Ont95wQRQrV6XrKGBl_yoJbSW5ey5yGo_631uGDe7lKdyZrX2vQdfkTLJncU1h8feJCSY5sWm_ixPoXYfaXllH9GyDfxyEI324GygiAt7Tt9; initiated 2022-03-01 10:04:59
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID w59XXReisdmN0ehzPUeM70XeBqpgEnkI3maMhn45OI.4tgSSbSEMqSjSZOT6YbQD7s8Z0xISg2XCCgwiuQvF6afQ05rc7K7ByhiONctu7k3mVWDcI2sMtMllqFG503Pv; initiated 2022-03-01 10:40:51
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID 2sq5FAgoyGTZhKCq9AMFLm7nWM9CAObNMp8T9uGX4v876s6Y0_l0Qct43tnaj10Wi.zbrN_JcpSzbQVyqluedBWY50825XiJMc1VWXKnMzQWG.o6MQjGKhmwwwoacx9R; initiated 2022-03-01 11:02:55
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID 7D_cKGWAwvY9gdAfNcxZaFFWeBbXcOfs0W4SOP0w7JXmyjseXwAhYDDNYjg.0SQLqtChnqJ3bnSxjEr436q1qlIz04na_e7mBIakRDh.ZE481q_81g.2bEMjS2eGonpZ; initiated 2022-03-01 11:09:39
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID 7VvAAjshvB4H_Du3wNHzUG.XaTmZZkMNuPbcqrWkv_G0Mbgxd_JgKQ_fcwWqc4QBybx0efMK86YQn61t.uG8bbc0p2593Fgj3yuf8SEYMLz4PlbVslIxylBt3.7fert3; initiated 2022-03-01 11:12:57
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID 5DZYym.HxS5GUCudes1vJPMankfJHo8blMT.L48etY_CgbtWYekPxrabP1tnNazLsHsiQ1VoFl0cBlELZjPaC9B30IkhBY8H2JA8.7iW5dI96FNy1RuZfHGpqbG87q4X; initiated 2022-03-01 11:45:56
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID 1GDMFxgOFLsA4OQhHcjU_9ddRjyU5g8YGEUnHJ8J0UBVzdHEaJU21h8jKeX79hoLRYa9ioMdtxs8RUOv8RfNUfzXy_asncnZyll2bzRhR4zbfdq6Whx0Ec4KawM4sbTY; initiated 2022-03-01 12:04:16
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID hLsdDJd54U.gAd6UCvvNmBmbf1uqOS35OzdveG6UX7IytHsXRbzXtdrkhTxClj9fB.Sa9l_p7X2LTA_jkJGWq1jS2FGDiTv.ejwreVbuE7ubW6pe8Lx44Ggqni_CdrSm; initiated 2022-03-01 13:38:55
Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID rIjsL3C9fy.RrbUwXwybfOjme79H93fANXIujvv1Irq5NxGBx_HBfvozCuYhU5AI2pNPJOLbdeBYmAo32nyYDZEoDWEy1FKRaji18k16bpMii4PEAxjxA6NFrtY7q6yC; initiated 2022-03-01 13:54:59 expected:<1> but was:<26>
[INFO]
[ERROR] Tests run: 117, Failures: 1, Errors: 0, Skipped: 9
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org