[D] Memory management for 2 input processor.

["Bauaser-kun (via GitHub)" <gi...@apache.org>]

GitHub user Bauaser-kun created a discussion: Memory management for 2 input processor.

Hi I've created a my first processor. The idea is pretty simple - take 2 input streams (total production & waste count) and output waste percentage from a batch.
THe processor uses the same stream buffer as OOTB merge 2 streams processor but as I can see the buffer keeps increaing in size with each event.
Is there any in-build memory management in event handler to not run out of memory? And if not is there any preffered way to efficiently clean buffer after an event/specified time?

GitHub link: https://github.com/apache/streampipes/discussions/2334

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Memory management for 2 input processor.

["Bauaser-kun (via GitHub)" <gi...@apache.org>]

GitHub user Bauaser-kun closed a discussion: Memory management for 2 input processor.

Hi I've created a my first processor. The idea is pretty simple - take 2 input streams (total production & waste count) and output waste percentage from a batch.
THe processor uses the same stream buffer as OOTB merge 2 streams processor but as I can see the buffer keeps increaing in size with each event.
Is there any in-build memory management in event handler to not run out of memory? And if not is there any preffered way to efficiently clean buffer after an event/specified time?

GitHub link: https://github.com/apache/streampipes/discussions/2334

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Memory management for 2 input processor.

["Bauaser-kun (via GitHub)" <gi...@apache.org>]

GitHub user Bauaser-kun added a comment to the discussion: Memory management for 2 input processor.

Sorry for the late response. As of know I don't have any suggestions, as for now we (as a company) are checking possibilites and more like playing with streampipes. We will however propably implement them internally and will discuss memory usage in bigger team. 

GitHub link: https://github.com/apache/streampipes/discussions/2334#discussioncomment-7894455

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Memory management for 2 input processor.

["tenthe (via GitHub)" <gi...@apache.org>]

GitHub user tenthe closed the discussion with a comment: Memory management for 2 input processor.

Thank you for your feedback. I would close the discussion. If you have any questions, please feel free to open it again. 
Also if there are any other questions please do not hesitate to contact us.

GitHub link: https://github.com/apache/streampipes/discussions/2334#discussioncomment-7895220

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Memory management for 2 input processor.

["tenthe (via GitHub)" <gi...@apache.org>]

GitHub user tenthe added a comment to the discussion: Memory management for 2 input processor.

Hello @Bauaser-kun,
thank you for the question. That is a good point. There is no built-in memory management.
The preferred way of handling state in memory depends on the use case.
In general, you should ensure that an appropriate cleanup strategy is in place.
However, since this is a common case, it would also be good to have a generic service that takes care of the state.
If you have a suggestion on how such a service could look like, post it here and we can discuss it.
What do you think of it?

GitHub link: https://github.com/apache/streampipes/discussions/2334#discussioncomment-7855023

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org