[jira] [Created] (HDDS-11939) Memory leak of DataStreamMapImpl

["ChenXi (Jira)" <ji...@apache.org>]

ChenXi created HDDS-11939:
-----------------------------

             Summary: Memory leak of DataStreamMapImpl
                 Key: HDDS-11939
                 URL: https://issues.apache.org/jira/browse/HDDS-11939
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: ChenXi
         Attachments: image-2024-12-15-21-14-27-612.png, image-2024-12-15-22-12-41-704.png, image-2024-12-15-22-14-22-904.png, image-2024-12-15-22-18-24-185.png, image-2024-12-15-22-21-26-187.png

h2. Phenomena

We have found a memory leak in DataStreamMapImpl when writing with Stream online.
This problem can be reproduced on the current master branch

DataStreamMapImpl even after the write request stops for a long time, will still hold a very large number of DataStream, these DataStream unless restarted, or more and more, can not be released

!image-2024-12-15-22-21-26-187.png!

!image-2024-12-15-21-14-27-612.png!
h2. Reproduction method
h3. Starting the cluster in Steaming mode

refer to: [https://ozone.apache.org/docs/edge/feature/streaming-write-pipeline.html]
h3. execute a command
{code:bash}
for i in `seq 1 10`; do ozone freon ommg --operation CREATE_STREAM_FILE -n 100 -t 100 --size=1M --volume s3v  --bucket bucket1 --duration 5;done
{code}
 
h3. h3. DataStreamMapImpl 

At this point in the DataStreamMapImpl has been left in some DataStream but because there is no log, so can not be directly observed, I added a log in the DataStreamMapImpl#remove to facilitate the observation, this is a screenshot of my logs

!image-2024-12-15-22-18-24-185.png!
h4.  
h3. h3. Netty's mem

And Netty's direct mem keeps growing, not dropping.

Before
!image-2024-12-15-22-12-41-704.png!

After
!image-2024-12-15-22-14-22-904.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org