You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by "andrewmusselman (via GitHub)" <gi...@apache.org> on 2026/09/03 01:24:03 UTC
[I] p6: backup for the LiteLLM database and fleet state (tooling-llmao)
andrewmusselman opened a new issue, #19:
URL: https://github.com/apache/tooling-llmao/issues/19
With fleet state in LiteLLM, there is one artifact rather than two.
```bash
sudo -u postgres pg_dump litellm | gzip > "$dest/litellm-$stamp.sql.gz"
```
**`$dest` is the open question.** Three constraints:
- the destination must be captured **off-host**. `bpc_client_asf` pulls via
rsync from a central BackupPC server, so the client has no local daemon and
coverage cannot be confirmed from the box. The share list lives on the
server — ask Infra which paths are pulled rather than assuming. A default
share of `/etc` and `/home` would miss `/var/backups`
- confirm the host is not in `bpc_client_asf::excludelist`
- a file-level copy of a live Postgres data directory is **not** a valid
backup; the `pg_dump` is what makes it restorable
What is at stake, and it grew with "LiteLLM routes have no api_base": Postgres now holds **routing** as well
as keys, budgets and spend. Losing it means the proxy no longer knows what to
proxy, not merely who may call it. Virtual keys are stored hashed and shown
once at mint, so recovery also means re-minting every key and reconfiguring
every consumer.
With fleet state in LiteLLM there is one artifact rather than two, and nothing
outside the database to reconstruct from. The GPU boxes keep serving from their
local assignment, so a lost database does not stop inference — but nothing can
re-fetch, re-route or re-authenticate until it is restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
Re: [I] p6: backup for the LiteLLM database and fleet state (tooling-llmao)
Posted by "gstein (via GitHub)" <gi...@apache.org>.
gstein commented on issue #19:
URL: https://github.com/apache/tooling-llmao/issues/19#issuecomment-5593462701
From @dfoulks1 , he wrote that examples for PG backups are located in these p6 modules:
`psql01-ec2-va, jiratest-ec2-va and jira2-he-de all have postgresql_asf::backup enabled.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
Re: [I] p6: backup for the LiteLLM database and fleet state (tooling-llmao)
Posted by "gstein (via GitHub)" <gi...@apache.org>.
gstein commented on issue #19:
URL: https://github.com/apache/tooling-llmao/issues/19#issuecomment-5593405701
Please use /x1/llmao for all data. That is the VM pattern that Infra uses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
Re: [I] p6: backup for the LiteLLM database and fleet state (tooling-llmao)
Posted by "gstein (via GitHub)" <gi...@apache.org>.
gstein commented on issue #19:
URL: https://github.com/apache/tooling-llmao/issues/19#issuecomment-5593540268
I don't recall where postgres (via APT install) puts its database. We likely don't care, as the pg_dump snapshots can go wherever we wish.
I did the pg install via APT. That would go into p6 `base_packages` or somesuch. The example modules above should have better info on how to ensure pg is installed on our VM. (rather than my manual install)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org