You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/04/19 14:50:10 UTC
[GitHub] [incubator-doris] caiconghui opened a new issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
caiconghui opened a new issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680
Now, we use time expire to control the history trasaction num and load job under db, but it is hard to make a balance between informaction viewing and memory usage, for some user may execute many load jobs such as insert job in a short time, which may cause heavy memory pressure which other user may execute only some load jobs for one day.
So we need to add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680#issuecomment-822954752
> @caiconghui insert-job, broker load job and transaction jobs could be checked by coders anytime .
> how about make them persistence, such as store in itself, make it a write audit log
this PR is mainly to control memory usage, and I think we can implement it in next PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui removed a comment on issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
Posted by GitBox <gi...@apache.org>.
caiconghui removed a comment on issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680#issuecomment-823989914
what is the format for audit log ? file or table data? I think if job writed in file, it is hard to managed, if stored in table, the steps may be some A bit cumbersome and data may be lost because we need to stream load it. In my option, we can do audit log by fe plugin as a addtiional way to record it while you can set expire time short
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680#issuecomment-823990047
> > > @caiconghui insert-job, broker load job and transaction jobs could be checked by coders anytime .
> > > how about make them persistence, such as store in itself, make it a write audit log
> >
> >
> > this PR is mainly to control memory usage, and I think we can implement it in next PR
>
> high memory usage cause of lots of history jobs in it , persistence such as store it in Doris or write it in audit log can
>
> * reduce memory usage.
> * be read any time by developpers for hisitory jobs.
>
> how do u like it ?
what is the format for audit log ? file or table data? I think if job writed in file, it is hard to managed, if stored in table, the steps may be some A bit cumbersome and data may be lost because we need to stream load it. In my option, we can do audit log by fe plugin as a addtiional way to record it while you can set expire time short
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] francisoliverlee commented on issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680#issuecomment-822943001
@caiconghui insert-job, broker load job and transaction jobs could be checked by coders anytime .
how about make them persistence, such as store in itself, make it a write audit log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680#issuecomment-823989914
what is the format for audit log ? file or table data? I think if job writed in file, it is hard to managed, if stored in table, the steps may be some A bit cumbersome and data may be lost because we need to stream load it. In my option, we can do audit log by fe plugin as a addtiional way to record it while you can set expire time short
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] francisoliverlee commented on issue #5680: [Config] Add config to limit the num of history load job num or history transaction under db to avoid too much memory usage
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #5680:
URL: https://github.com/apache/incubator-doris/issues/5680#issuecomment-823801059
> > @caiconghui insert-job, broker load job and transaction jobs could be checked by coders anytime .
> > how about make them persistence, such as store in itself, make it a write audit log
>
> this PR is mainly to control memory usage, and I think we can implement it in next PR
high memory usage cause of lots of history jobs in it , persistence such as store it in Doris or write it in audit log can
- reduce memory usage.
- be read any time by developpers for hisitory jobs.
how do u like it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org