You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Krishna Kumar Asawa (Jira)" <ji...@apache.org> on 2023/08/02 09:12:00 UTC
[jira] [Assigned] (HDDS-6225) Pipelines distribution across datanodes is uneven
[ https://issues.apache.org/jira/browse/HDDS-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Kumar Asawa reassigned HDDS-6225:
-----------------------------------------
Assignee: Nandakumar
> Pipelines distribution across datanodes is uneven
> -------------------------------------------------
>
> Key: HDDS-6225
> URL: https://issues.apache.org/jira/browse/HDDS-6225
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Maksim Myskov
> Assignee: Nandakumar
> Priority: Major
>
> Despite HDDS-5441 has been done to improve pipeline distribution, there is still an issue.
> I made a couple of tests:
> Configuration1. 6 datanodes, ozone.scm.datanode.disallow.same.peers = true, 3 pipelines
> || Datanode||Pipelines||
> |8cc97cc8-31dd-463d-8a89-7c76d95d709f|2|
> |ab3dc657-736e-49a0-a5c9-e0e4962773d9|2|
> |c3f91923-305f-48b1-b38b-2db1e4b92162|2|
> |cd16db96-92cb-4056-86f8-563e1ac1bfce|2|
> |292584d1-6de5-4249-a601-25a813df997e|2|
> |4bf6cae2-1757-4b67-aba4-272d3148c838|2|
>
> Configuration2. 6 datanodes, ozone.scm.datanode.disallow.same.peers = false, 8 pipelines
> ||Datanode||Pipelines||
> |cc131824-ccbe-4e8a-9ec4-e7889275e90b|5|
> |164b4515-7735-4fb5-9836-d995c9fafb88|5|
> |1a217fc9-9b45-4161-b1e8-fda71a3c38c6|5|
> |2f0bcc4e-407a-4e88-8465-d9e5e6ba8bdc|3|
> |5df86a41-3c17-4dc1-9e57-410fe5ad7842|1|
> |71db1b20-5d89-4919-8566-683c3f24ad5e|5|
>
> Configuration3. 7 datanodes, ozone.scm.datanode.disallow.same.peers = true, 7 pipelines
> ||Datanode||Pipelines||
> |a630e56a-3e1c-47c3-8d49-fb6636a81f2f|3|
> |b99a61c9-d907-4536-ac65-e1df254b6190|3|
> |005f3c5a-5001-46ef-9016-44114e7b4e32|3|
> |213d0cfd-afbb-462e-9f70-2b959a2dbe91|3|
> |6531b78c-2ba0-44c2-a7f7-7271f47bd381|3|
> |706625cb-15ab-4f5e-a23a-f2de72946490|3|
> |77ed9147-6a86-43e1-81fe-ed2378f8f518|3|
>
> Configuration4. 7 datanodes, ozone.scm.datanode.disallow.same.peers = false, 10 pipelines
> ||Datanode||Pipelines||
> |9a019bd1-fa44-4e67-950f-d2e15338b0cd|5|
> |c948417a-784e-4dd2-8ca3-715d67ac9891|5|
> |d49c9c19-1a60-407e-975b-9fe00a8ac44c|5|
> |e2792111-0506-49ed-8096-6bed67675fc4|5|
> |0fb9ddcd-5bda-4a8b-bada-39f6f2ec7ec2|5|
> |2011f197-6248-4a6f-9c7b-e3611e67d3d0|5|
> |782d9313-345c-4297-ab6b-d57b7eb4910f|0|
>
> Configuration5. 8 datanodes, ozone.scm.datanode.disallow.same.peers = true, 7 pipelines
> ||Datanode||Pipelines||
> |9b1146e2-7ff0-4e25-b42d-7e7217bd01f3|3|
> |a593ecab-5237-4c0e-bcd0-411839bcd1ef|3|
> |02f2c58a-c172-4e10-84e9-5cae850327f9|3|
> |0f18f35a-56a2-42ea-8f0e-7d2a84b16c6c|3|
> |3a11728e-637f-4337-a3cf-7b9a274e3614|3|
> |446a32f2-fdb3-4386-813c-0e5f1c9f8c31|0|
> |56522197-646a-4964-8e94-d9c8f7e8687b|3|
> |7f21c5a2-fcd0-4b40-8b0c-cc647c50e6b1|3|
>
> Configuration6. 8 datanodes, ozone.scm.datanode.disallow.same.peers = false 12 pipelines
> ||Datanode||Pipelines||
> |9eba6024-9c1e-4a63-b0ea-c71593d0efeb|5|
> |ad6ff8f0-336f-4ba5-b203-787870d1b7ca|5|
> |030f3e80-6ac1-4c87-ab25-6538b50602cd|5|
> |09e83e1d-8e9b-429a-8f89-81cba1523487|5|
> |226b0722-42f1-47e4-8b90-083e7174a920|5|
> |459a8649-4238-407d-9d3e-4d0577acf496|5|
> |4680966d-fd24-4f24-831d-084382c93dfe|5|
> |4aacc584-138c-4ce3-bb61-64bca8384571|1|
>
> Configuration7. 9 datanodes, ozone.scm.datanode.disallow.same.peers = true, 8 pipelines
> ||Datanode||Pipelines||
> |a22a1248-0627-4616-a896-3d434687b4c9|1|
> |b9cc0695-4817-4f7d-ab70-b2257bfb3fc4|3|
> |bfd99d04-897b-4695-a328-31195bed7a46|4|
> |fefa8454-a79e-4137-95fa-90f1942b5b1c|1|
> |0c4a3312-7313-4b23-a5fa-f6bc2f7f1593|3|
> |0e8f830e-40b0-46fb-b0f8-8bb07447b816|3|
> |490cd956-16d6-401b-af54-9f2570dc044c|3|
> |517346bc-4f3a-4c65-9141-0fb9b580bf74|3|
> |5ec043f7-b69f-4780-8383-e085569ddc96|3|
>
> Configuratio8. 9 datanodes, ozone.scm.datanode.disallow.same.peers = false, 15 pipelines
> ||Datanode||Pipelines||
> |a31576cf-1ff7-4931-b4f1-561e26deefff|5|
> |c8ecddeb-1c6e-4f35-b4c7-a535ed95c53f|5|
> |fae64881-b815-4fb1-ac8a-6932f7792d69|5|
> |13fdcaa1-6032-4fc9-946b-7e92577cbd32|5|
> |2024dce3-2de3-416b-bf62-2b4dff54cd9f|5|
> |38e6915a-90e7-444f-a6f3-72b0b03de29d|5|
> |60206eff-6630-4640-b68b-f3986bf3d299|5|
> |68040769-eab6-49bc-a5b4-ebe179d0df09|5|
> |7f755ac1-2fcd-48f6-8c18-b82914955f58|5|
>
> {color:#000000}Configuration9. 10 {color} datanodes, ozone.scm.datanode.disallow.same.peers = true, 11 pipelines
> ||Datanode||Pipelines||
> |8edb6712-e5c2-4f19-a9d2-be80cb444ddd|4|
> |9676338d-125a-4798-98ad-1dd25fbbf150|2|
> |d4607619-925a-4d26-85b1-7fb29fdd276a|4|
> |d72313a7-4d95-44f8-b020-716ea44d7a24|3|
> |daa01019-1595-4951-a404-c7ef88377280|4|
> |f26d824a-40e1-4fef-be74-13021ec3383c|4|
> |122d4373-16df-406e-b937-07a136031c98|3|
> |26b84595-9bb3-40b5-8b19-f678de2f7ed3|4|
> |28f8a99d-52e5-429e-b5a1-6cb3f180b2fa|2|
> |44d41c19-8f30-48b4-a466-8d8413f21705|3|
>
> {color:#000000}Configuration10. 10 {color} datanodes, ozone.scm.datanode.disallow.same.peers = false, 16 pipelines
> ||Datanode||Pipelines||
> |a5b0d0a8-41f0-4110-aef1-73f7a63defa2|5|
> |a9600399-19c6-4a4d-bd98-73853807266|5|
> |fcefa984-e74f-4370-8bb7-e13fddd549fc|5|
> |14a2cd7e-1f10-45f5-8ba0-d6515f6031b|5|
> |1fc1c312-c9b7-403f-aa42-db44d07d4812|5|
> |27d5af80-0bfe-423e-9b9d-882c380c5b28|3|
> |423c5624-683b-4269-9444-b01bc69093a|5|
> |5de9099e-0d3d-47a3-a9d2-344f99efaf7c|5|
> |5edd32e1-ea56-4485-94cd-42ada48211da|5|
> |64bc438e-a00d-4b62-9a62-2e5496058468|5|
>
> Take a look at configuration5 and configuration6. In this case property introduced in HDDS-5441 makes worse.
> Sometimes it helps(configuration4), sometimes not.
>
> The root cause of the issue seems to be lack of pipeline rebalancing mechanism. If pipeline distribution of a cluster is uneven, I can manually rebalance pipelines by closing existing pipelines (via ozone admin tool), in this case SCM will create new pipelines instead of closed ones. It would be great for SCM to rebalance pipelines automatically (including cases when a new node is added to a cluster). Does such rebalancing mechanism fit Ozone architecture? Or is there another way to avoid uneven pipelines distribution?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org