You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2024/12/04 21:08:17 UTC
[I] Change defaults on bulk import from unlimited. [accumulo]
keith-turner opened a new issue, #5138:
URL: https://github.com/apache/accumulo/issues/5138
**Is your feature request related to a problem? Please describe.**
Bulk imports that continually add files to a tablet that already has a lot of files will eventually cause problems.
**Describe the solution you'd like**
#5104 adds limits that can prevent this situation from happening, but those default to unlimited. There is also an existing property table.bulk.max.tablets that was introduced in 2.1.0 that defaults to unlimited. After #5104 is merged could change those defaults to a limit and also change the existing property table.bulk.max.tablets to also have a default limit. Could set the limit of all these properties to 100 to start with. When a user encounters the limit they can adjust as needed. This will make the default behavior of the system more sustainable.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Change defaults on bulk import from unlimited. [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr closed issue #5138: Change defaults on bulk import from unlimited.
URL: https://github.com/apache/accumulo/issues/5138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Change defaults on bulk import from unlimited. [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #5138:
URL: https://github.com/apache/accumulo/issues/5138#issuecomment-2518569614
https://github.com/apache/accumulo/issues/5117#issuecomment-2518542540 this is an example of one problem that could happen w/ too many files in tablet. Most Accumulo code assumes that all tablet metadata can be read into memory. When a tablet has 25K files and many threads reading that tablets metadata into memory it can cause a lot of memory pressure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org