You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2023/03/04 07:25:32 UTC
[GitHub] [doris] morningman opened a new issue, #17413: [Feature] Refresh catalog' metadata at fix interval.
morningman opened a new issue, #17413:
URL: https://github.com/apache/doris/issues/17413
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
## Background
Doris supports [multi catalog ](https://doris.apache.org/docs/dev/lakehouse/multi-catalog/) to connect to external datasource.
For any catalog, Doris will sync the metadata, such as database, table, schema, partition, file from external datasource
into FE's memory, so that when accessing the catalog, we don' t need to access the remote data source' meta service.
But it is a one-time operation, after sync, if the external data source change its metadata, such as adding new table, partition,
Doris can't be aware of it and the newly created metadata can not be accessed from Doris.
So we need a way refresh the metadata of the catalog, to sync the metadata at fix interval.
## How to
First, we should set new properties of catalog when creating it, eg:
```
create catalog hive properties (
...
"metadata_refresh_interval_sec" = "7200"
...
);
```
which means for this `hive` catalog, its metadata will be refreshed every 7200s.
And there will be a daemon thread on Master FE to do this for all catalogs.
## Tips
1. Related classes: See `makeSureInitialized()` of `ExternalCatalog` class to learn how to refresh catalog.
2. See [Refresh](https://doris.apache.org/docs/dev/sql-manual/sql-reference/Utility-Statements/REFRESH) command to learn how to refresh catalog manually.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] Refresh catalog' metadata at fix interval. [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed issue #17413: [Feature] Refresh catalog' metadata at fix interval.
URL: https://github.com/apache/doris/issues/17413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org