You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Feng Jin <ji...@gmail.com> on 2023/06/01 03:25:03 UTC
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Hi ， thanks all for reviewing the flip.
@Ron
> Regarding the CatalogStoreFactory#createCatalogStore method, do we need
to provide a default implementation?
Yes, we will provide a default InMemoryCatalogStoreFactory to create an
InMemoryCatalogStore.
> If we get a Catalog from CatalogStore, after initializing it, whether we
put it in Map<String, Catalog> catalogs again?
Yes, in the current design, catalogs are stored as snapshots, and once
initialized, the Catalog will be placed in the Map<String, Catalog>
catalogs.
Subsequently, the Map<String, Catalog> catalogs will be the primary source
for obtaining the corresponding Catalog.
> how about renaming them to `catalog.store.type` and
`catalog.store.path`?
I think it is okay. Adding "sql" at the beginning may seem a bit strange. I
will update the FLIP.
@Shammon
Thank you for the review. I have made the necessary corrections.
Regarding the modifications made to the Public Interface, I have also
included the relevant changes to the `TableEnvironment`.
Best,
Feng
On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
> Hi feng,
>
> Thanks for updating, I have some minor comments
>
> 1. The modification of `CatalogManager` should not be in `Public
> Interfaces`, it is not a public interface.
>
> 2. `@PublicEvolving` should be added for `CatalogStore` and
> `CatalogStoreFactory`
>
> 3. The code `Optional<Catalog> optionalDescriptor =
> catalogStore.get(catalogName);` in the `CatalogManager` should be
> `Optional<CatalogDescriptor> optionalDescriptor =
> catalogStore.get(catalogName);`
>
> Best,
> Shammon FY
>
>
> On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
>
> > Hi, Feng
> >
> > Thanks for driving this FLIP, this proposal is very useful for catalog
> > management.
> > I have some small questions:
> >
> > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> need
> > to provide a default implementation?
> > 2. If we get Catalog from CatalogStore, after initializing it, whether we
> > put it to Map<String, Catalog> catalogs again?
> > 3. Regarding the options `sql.catalog.store.type` and
> > `sql.catalog.store.file.path`, how about renaming them to
> > `catalog.store.type` and `catalog.store.path`?
> >
> > Best,
> > Ron
> >
> > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> >
> > > Hi yuxia
> > >
> > > > But from the code in Proposed Changes, once we register the Catalog,
> > we
> > > initialize it and open it. right?
> > >
> > > Yes, In order to avoid inconsistent semantics of the original CREATE
> > > CATALOG DDL, Catalog will be directly initialized in registerCatalog so
> > > that parameter validation can be performed.
> > >
> > > In the current design, lazy initialization is mainly reflected in
> > > getCatalog. If CatalogStore has already saved some catalog
> > configurations,
> > > only initialization is required in getCatalog.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > wrote:
> > >
> > > > Hi, Feng.
> > > > I'm trying to understanding the meaning of *lazy initialization*. If
> > i'm
> > > > wrong, please correct me.
> > > >
> > > > IIUC, lazy initialization means only you need to access the catalog,
> > then
> > > > you initialize it. But from the code in Proposed Changes, once we
> > > register
> > > > the Catalog,
> > > > we initialize it and open it. right?
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > ----- 原始邮件 -----
> > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > 收件人: "dev" <de...@flink.apache.org>
> > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > configuration
> > > > and asynchronous registration
> > > >
> > > > Hi Feng,
> > > >
> > > > Thanks for your effort! +1 for the proposal.
> > > >
> > > > One of the major changes is that current design will provide
> > > > Map<String,Catalog> catalogs as a snapshot instead of a cache, which
> > > means
> > > > once it has been initialized, any changes done by other sessions will
> > not
> > > > affect it. Point 6 described follow-up options for further
> improvement.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >
> > > > > Hi all, I would like to update you on the latest progress of the
> > FLIP.
> > > > >
> > > > >
> > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang
> > > and I
> > > > > had an offline discussion regarding the overall solution for Flink
> > > > > CatalogStore. We have reached a consensus and I have updated the
> > final
> > > > > solution in FLIP.
> > > > >
> > > > > Next, let me briefly describe the entire design:
> > > > >
> > > > > 1.
> > > > >
> > > > > Introduce CatalogDescriptor to store catalog configuration
> similar
> > > to
> > > > > TableDescriptor.
> > > > > 2.
> > > > >
> > > > > The two key functions of CatalogStore - void storeCatalog(String
> > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > getCatalog(String)
> > > > > will both use CatalogDescriptor instead of Catalog instance.
> This
> > > way,
> > > > > CatalogStore will only be responsible for saving and retrieving
> > > > catalog
> > > > > configurations without having to initialize catalogs.
> > > > > 3.
> > > > >
> > > > > The default registerCatalog(String catalogName, Catalog catalog)
> > > > > function in CatalogManager will be marked as deprecated.
> > > > > 4.
> > > > >
> > > > > A new function registerCatalog(String catalogName,
> > CatalogDescriptor
> > > > > catalog) will be added to serve as the default registration
> > function
> > > > for
> > > > > catalogs in CatalogManager.
> > > > > 5.
> > > > >
> > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > unchanged
> > > > and
> > > > > save initialized catalogs.This means that deletion operations
> from
> > > one
> > > > > session won't synchronize with other sessions.
> > > > > 6.
> > > > >
> > > > > To support multi-session synchronization scenarios for deletions
> > > later
> > > > > on we should make Map<String,Catalog>catalogs configurable.There
> > may
> > > > be
> > > > > three possible situations:
> > > > >
> > > > > a.Default caching of all initialized catalogs
> > > > >
> > > > > b.Introduction of LRU cache logic which can automatically clear
> > > > > long-unused catalogs.
> > > > >
> > > > > c.No caching of any instances; each call to getCatalog creates a
> > new
> > > > > instance.
> > > > >
> > > > >
> > > > > This is the document for discussion:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > >
> > > > > This is the final proposal document:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > >
> > > > >
> > > > > Thank you very much for your attention and suggestions on this
> > FLIP. A
> > > > > special thanks to Hang Ruan for his suggestions on the entire
> design
> > > and
> > > > > organizing offline discussions.
> > > > >
> > > > > If you have any further suggestions or feedback about this FLIP
> > please
> > > > feel
> > > > > free to share.
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Feng
> > > > >
> > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge <jing@ververica.com.invalid
> >
> > > > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for improving the FLIP. It looks good to me. We could
> still
> > > > > > reconsider in the future how to provide more common built-in
> cache
> > > > > > functionality in CatalogManager so that not every CatalogSotre
> > > > > > implementation has to take care of it.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Jing,
> > > > > > >
> > > > > > > Thanks for your reply.
> > > > > > >
> > > > > > > > There might be more such issues. I would suggest you
> > completely
> > > > walk
> > > > > > > through the FLIP again and fix those issues
> > > > > > >
> > > > > > > I am very sorry for my carelessness and at the same time, I
> > greatly
> > > > > > > appreciate your careful review. I have thoroughly checked the
> > > entire
> > > > > FLIP
> > > > > > > and made corrections to these issues.
> > > > > > >
> > > > > > >
> > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > CatalogManager
> > > > > > > could work without Optional CatalogStore being configured.
> > > > > > >
> > > > > > > Yes, in the original design, CatalogStore was not necessary
> > because
> > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > catalog
> > > > > > > instances.
> > > > > > > However, this caused inconsistency issues. Therefore, I
> modified
> > > this
> > > > > > part
> > > > > > > of the design and removed Map<String, Catalog> catalogs from
> > > > > > > CatalogManager.
> > > > > > > At the same time, InMemoryCatalog will serve as the default
> > > > > CatalogStore
> > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > Map<String,Catalog>catalogs.
> > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> been
> > > > moved
> > > > > to
> > > > > > > Rejected Alternatives.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > There are still many places contain inconsistent content,
> e.g.
> > > > > > > >
> > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > method
> > > is
> > > > > > > doing.
> > > > > > > >
> > > > > > > >
> > > > > > > > There might be more such issues. I would suggest you
> completely
> > > > walk
> > > > > > > > through the FLIP again and fix those issues.
> > > > > > > >
> > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> build
> > > the
> > > > > > cache
> > > > > > > > functionality in the CatalogStore? This is a very different
> > > design
> > > > > > > concept
> > > > > > > > from what the current FLIP described. If I am not mistaken,
> > with
> > > > the
> > > > > > > > current FLIP design, CatalogManager could work without
> Optional
> > > > > > > > CatalogStore being configured. That is the reason why I
> > mentioned
> > > > in
> > > > > > the
> > > > > > > > last email that the example code wrt the Optional
> CatalogStore
> > is
> > > > not
> > > > > > > > correct.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > jinfeng1995@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > >
> > > > > > > > > Thank you very much for the careful review. I have already
> > made
> > > > the
> > > > > > > > > relevant changes.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > scenarios, I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > sure if this is the intended behaviour
> > > > > > > > >
> > > > > > > > > Currently, there are conflicts in multiple scenarios with
> the
> > > > > current
> > > > > > > > > design. I am thinking whether we should remove 'Map<String,
> > > > > Catalog>'
> > > > > > > and
> > > > > > > > > make Cache the default behavior of InMemoryCatalogStore.
> This
> > > > way,
> > > > > > > users
> > > > > > > > > can implement their own CatalogStore to achieve
> > multi-instance
> > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for working on the FLIP. There are still some NIT
> > > issues
> > > > > in
> > > > > > > the
> > > > > > > > > FLIP
> > > > > > > > > > like:
> > > > > > > > > >
> > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > > > > CatalogStore
> > > > > > > > > > instead of Optional in the code example. It should be
> fine
> > to
> > > > use
> > > > > > it
> > > > > > > as
> > > > > > > > > > pseudo code for now and update it after you submit the
> PR.
> > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> rejected
> > > > > section
> > > > > > > > which
> > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > >
> > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > scenarios,
> > > > > I
> > > > > > am
> > > > > > > > not
> > > > > > > > > > sure if this is the intended behaviour. If Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > is
> > > > > > > > > > used as a cache, it should be invalid, once the related
> > > catalog
> > > > > has
> > > > > > > > been
> > > > > > > > > > removed from the CatalogStore by another instance. Did I
> > miss
> > > > > > > > something?
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > jinfeng1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > >
> > > > > > > > > > > @Jing
> > > > > > > > > > >
> > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > I think this is a good function name, I have updated it
> > in
> > > > the
> > > > > > > > > > > documentation.
> > > > > > > > > > >
> > > > > > > > > > > > Some common cache features should be implemented
> > > > > > > > > > > Thank you for the suggestion. If alternative 1 turns
> out
> > to
> > > > be
> > > > > > more
> > > > > > > > > > > appropriate later, I will improve this part of the
> > content.
> > > > > > > > > > >
> > > > > > > > > > > > As the above discussion moves forward, the option 2
> > > > solution
> > > > > > > looks
> > > > > > > > > more
> > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > Yes, after discussing with Shammon offline, we think
> that
> > > > > > solution
> > > > > > > 2
> > > > > > > > > > might
> > > > > > > > > > > be more suitable and also avoid any inconsistency
> issues.
> > > > > > > > > > >
> > > > > > > > > > > > There are some inconsistent descriptions in the
> > content.
> > > > > Would
> > > > > > > you
> > > > > > > > > > like
> > > > > > > > > > > to clean them up?
> > > > > > > > > > > I will do my best to improve the document and
> appreciate
> > > your
> > > > > > > > > > suggestions.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > @Shammon
> > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > Alternatives`
> > > > > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > zjureel@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > >
> > > > > > > > > > > > I found there are two options in `Proposed Changes`,
> > can
> > > > you
> > > > > > put
> > > > > > > > the
> > > > > > > > > > > > unselected option in `Rejected Alternatives`? I think
> > > this
> > > > > may
> > > > > > > help
> > > > > > > > > us
> > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Shammon FY
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for raising this FLIP. I am still confused
> > after
> > > > > > > > completely
> > > > > > > > > > > > reading
> > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > addCatalog()
> > > > > have
> > > > > > > no
> > > > > > > > > big
> > > > > > > > > > > > > difference based on their names. One of them is
> > > > responsible
> > > > > > for
> > > > > > > > > data
> > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > catalogs
> > > is
> > > > > > used
> > > > > > > > as a
> > > > > > > > > > > cache
> > > > > > > > > > > > > and catalogStore is used for data persistence. I
> > would
> > > > > > suggest
> > > > > > > > > > > describing
> > > > > > > > > > > > > their purpose conceptually and clearly in the FLIP.
> > > Some
> > > > > > common
> > > > > > > > > cache
> > > > > > > > > > > > > features should be implemented, i.e. data in the
> > cache
> > > > and
> > > > > > the
> > > > > > > > > store
> > > > > > > > > > > > should
> > > > > > > > > > > > > be consistent. Same Catalog instance should be
> found
> > in
> > > > the
> > > > > > > store
> > > > > > > > > and
> > > > > > > > > > > in
> > > > > > > > > > > > > the cache(either it has been initialized or it will
> > be
> > > > lazy
> > > > > > > > > > > initialized)
> > > > > > > > > > > > > for the same catalog name. The consistency will be
> > > taken
> > > > > care
> > > > > > > of
> > > > > > > > > > while
> > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > 3. As the above discussion moves forward, the
> option
> > 2
> > > > > > solution
> > > > > > > > > looks
> > > > > > > > > > > > more
> > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > issues
> > > > > > > mentioned
> > > > > > > > > > > > > previously with option 1 are not solved yet. Do you
> > > still
> > > > > > want
> > > > > > > to
> > > > > > > > > > > propose
> > > > > > > > > > > > > both options and ask for suggestions for both of
> > them?
> > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > inconsistent
> > > > > > > > > > descriptions
> > > > > > > > > > > > in
> > > > > > > > > > > > > the content. Would you like to clean them up?
> > Thanks!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you for your response, and I completely
> agree
> > > > with
> > > > > > your
> > > > > > > > > point
> > > > > > > > > > > of
> > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > Initially, I may have over complicated the whole
> > > issue.
> > > > > > First
> > > > > > > > and
> > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > we need to consider the persistence of the
> > Catalog's
> > > > > > > > > Configuration.
> > > > > > > > > > > > > > If we only need to provide persistence for
> Catalog
> > > > > > > > Configuration,
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > interface.
> > > > > > > > > > > > > > This method can convert a Catalog instance to a
> > > > > Map<String,
> > > > > > > > > String>
> > > > > > > > > > > > > > properties, and the default implementation will
> > throw
> > > > an
> > > > > > > > > exception.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > * Returns a map containing the properties of
> the
> > > > > catalog
> > > > > > > > > object.
> > > > > > > > > > > > > > *
> > > > > > > > > > > > > > * @return a map containing the properties of
> the
> > > > > catalog
> > > > > > > > > object
> > > > > > > > > > > > > > * @throws UnsupportedOperationException if the
> > > > > > > implementing
> > > > > > > > > > class
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > > > > */
> > > > > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > > > > throw new
> UnsupportedOperationException("Please
> > > > > > implement
> > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. If the user has configured a CatalogStore,
> the
> > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > will be called when registering a Catalog
> instance
> > > with
> > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > catalog).
> > > > The
> > > > > > > > Catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > will be converted to a Map<String, String>
> > properties
> > > > and
> > > > > > > saved
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > CatalogStore. If some Catalog instances do not
> > > > implement
> > > > > > this
> > > > > > > > > > method,
> > > > > > > > > > > > an
> > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. If the user does not use a CatalogStore, the
> > > > > > > > > toConfiguration()
> > > > > > > > > > > > method
> > > > > > > > > > > > > > will not be called, ensuring consistency with the
> > > > > original
> > > > > > > > > process.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog> catalogs
> > and
> > > > the
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > at
> > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > we can start from the Catalog itself and provide
> a
> > > > > > dedicated
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > ConfigurationCatalog
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > Catalog
> > > {
> > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > properties)
> > > {
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > > }
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > zjureel@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have no questions about the functionality of
> > > > > > > > `CatalogStore`,
> > > > > > > > > > but
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > different operations of multiple
> > `registerCatalog`
> > > or
> > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > `CatalogManager`. The implementation in Trino
> is
> > > also
> > > > > the
> > > > > > > > same,
> > > > > > > > > > the
> > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > `createCatalog`,
> > > > > > > which
> > > > > > > > > > will
> > > > > > > > > > > > save
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think we don't need to add another
> > > > `registerCatalog`
> > > > > or
> > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> catalogs`
> > > or
> > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > > > > `Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How
> > about
> > > > > > saving
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> `CatalogStore
> > > > > > > > catalogStore`
> > > > > > > > > > > > > together
> > > > > > > > > > > > > > > when it is registered or added in
> > `CatalogManager`?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry for
> > the
> > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > by my deviation from the original discussion.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > this
> > > > FLIP
> > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> before.
> > > > > > However,
> > > > > > > > in
> > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > CatalogManager
> > > > > > > > > itself
> > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > need to change much. *We cannot only persist
> > > > Catalog
> > > > > > > > > instances
> > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > not be enough to save Catalogs, because in
> the
> > > end
> > > > we
> > > > > > > still
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > the configurations corresponding to the
> Catalog
> > > > > > > instances.
> > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> interface
> > > for
> > > > > > > > > > configuration
> > > > > > > > > > > > > > > > persistence. Regarding this part, I also took
> > > > > > inspiration
> > > > > > > > > from
> > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> the
> > > > same
> > > > > > name
> > > > > > > > > > looks a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > Thank you for the suggestion. I will provide
> a
> > > > > detailed
> > > > > > > > > > > description
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > code examples for this part, and add it to
> the
> > > > FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog` still
> > > looks
> > > > > > > confused
> > > > > > > > > to
> > > > > > > > > > > me.
> > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > this method is for storing catalog to
> > > CatalogStore,
> > > > > how
> > > > > > > > about
> > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> opinion,
> > > you
> > > > > can
> > > > > > > > decide
> > > > > > > > > > to
> > > > > > > > > > > > take
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I
> > don't
> > > > see
> > > > > > any
> > > > > > > > > > problem
> > > > > > > > > > > > with
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> method
> > > will
> > > > > be
> > > > > > > > > called?
> > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > wondering
> > > > > > whether
> > > > > > > > user
> > > > > > > > > > can
> > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > For CREATE CATALOG, my original design was to
> > add
> > > > it
> > > > > > > > directly
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > existing
> > > > > > logic.
> > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > we can do both: save the configuration to the
> > > > > > > CatalogStore
> > > > > > > > > and
> > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > default
> > > > > > > > > implmentation
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > I think it is necessary, otherwise we would
> > need
> > > to
> > > > > > > > introduce
> > > > > > > > > > an
> > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > loading.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > catalog.
> > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > made into an asynchronous interface because
> > > Catalog
> > > > > is
> > > > > > > > > already
> > > > > > > > > > an
> > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It looks more like lazy initialization for
> > > > catalog
> > > > > > than
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > should
> > > be
> > > > > > lazy
> > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > instead of asynchronous registration. I have
> > > > already
> > > > > > > > updated
> > > > > > > > > > the
> > > > > > > > > > > > > title
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > > > > zjureel@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > method
> > > in
> > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > confuse users whether a method named
> > > `addCatalog`
> > > > > or
> > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > And as you mentioned, the memory catalog
> is a
> > > > > > `cache`,
> > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > of `cache` should not be exposed to users.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > this
> > > > > FLIP.
> > > > > > > > Please
> > > > > > > > > > > > correct
> > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > discussion
> > > > > is
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > implement
> > > it
> > > > > for
> > > > > > > > > > different
> > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think there is a gap between the current
> > FLIP
> > > > > > design
> > > > > > > > and
> > > > > > > > > > that
> > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > What about the proposal of the discussion
> in
> > > > thread
> > > > > > > [1] ?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > >
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> > the
> > > > > same
> > > > > > > name
> > > > > > > > > > > looks a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > complex to me. I think it'll be better to
> > > > explain
> > > > > > it
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > java
> > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > these methods and give a brief example in
> > > this
> > > > > > FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> still
> > > > looks
> > > > > > > > confused
> > > > > > > > > > to
> > > > > > > > > > > > me.
> > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > CatalogStore,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > opinion,
> > > > > you
> > > > > > > can
> > > > > > > > > > decide
> > > > > > > > > > > > to
> > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which
> > method
> > > > > will
> > > > > > be
> > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > wondering
> > > > > > > > whether
> > > > > > > > > > user
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3: Is it really neccessary to provide a
> > > default
> > > > > > > > > > implmentation
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > catalog.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > CATALOG,
> > > > > the
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > > > > If asynchronous registration is the
> default
> > > > > > behavior,
> > > > > > > > it
> > > > > > > > > > > there
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > registration
> > > > > > just
> > > > > > > > > like
> > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > `registerCatalog`
> > > > > > > be
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration, it
> may
> > > well
> > > > > > that
> > > > > > > > > CREATE
> > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > executes successfully, but then the
> > following
> > > > > > CREATE
> > > > > > > > > TABLE
> > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > I think it's a break change which should
> be
> > > > > > > highlighted
> > > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> registration, I
> > > > would
> > > > > > > like
> > > > > > > > to
> > > > > > > > > > > > expect
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > will be an executor to open or register
> > > catalog
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > "the Catalog instance will be initialized
> > if
> > > it
> > > > > has
> > > > > > > not
> > > > > > > > > > been
> > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > yet. If the initialization process fails,
> > > these
> > > > > > > > > statements
> > > > > > > > > > > will
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> for
> > > > > catalog
> > > > > > > than
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <ji...@gmail.com>
> > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > persistence
> > > > > of
> > > > > > > > > Catalog
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > interest
> > > in
> > > > > > this
> > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > How to handle a catalog with the same
> > name
> > > > that
> > > > > > > > exists
> > > > > > > > > > for
> > > > > > > > > > > > both
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I believe this is a crucial point. Based
> on
> > > my
> > > > > > > current
> > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> > > > catalogs
> > > > > > will
> > > > > > > > > serve
> > > > > > > > > > > as
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > for instantiated catalogs and have the
> > > highest
> > > > > > > > priority.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are three methods that can affect
> the
> > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method puts the catalog instance
> into
> > > the
> > > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String catalogName)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method removes the catalog instance
> > > > > > > corresponding
> > > > > > > > to
> > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method first retrieves the
> > corresponding
> > > > > > catalog
> > > > > > > > > > > instance
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If the
> > > > catalog
> > > > > > > does
> > > > > > > > > not
> > > > > > > > > > > > exist,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > retrieves the corresponding configuration
> > > from
> > > > > the
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > initializes it, and puts the initialized
> > > > Catalog
> > > > > > > > instance
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The following two methods only modify the
> > > > > > > configuration
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > Map<String,
> > > > > > String>
> > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This method saves the properties to the
> > > > > > catalogStore
> > > > > > > > and
> > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > whether there is a catalogName with the
> > same
> > > > > name.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > This method removes the specified
> > > configuration
> > > > > of
> > > > > > > the
> > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The following are possible conflict
> > > scenarios:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > already
> > > > > > exists
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> > > > Catalog>,
> > > > > > the
> > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> Catalog
> > > > > > catalog)
> > > > > > > > > method
> > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > already
> > > > > > exists
> > > > > > > in
> > > > > > > > > > both
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> Catalog>,
> > > the
> > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method will
> > > fail.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > already
> > > > > > exists
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> addCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > String> properties) method can directly
> > save
> > > > the
> > > > > > > > > properties
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > > > > catalogName)
> > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > use the new properties for initialization
> > > > because
> > > > > > the
> > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > catalog instance already exists in
> catalogs
> > > and
> > > > > > will
> > > > > > > be
> > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > Therefore, using the
> > unregisterCatalog(String
> > > > > > > > > catalogName)
> > > > > > > > > > > > method
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > remove the instance corresponding to the
> > > > original
> > > > > > > > > > catalogName
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > `Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > Map<String,
> > > > > > > > String>
> > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This could potentially lead to
> confusion. I
> > > > > suggest
> > > > > > > > > > changing
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > method name, perhaps to addCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > String> properties), as previously
> > mentioned
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > catalogName,Catalog
> > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Since a catalog is already an instance,
> > > adding
> > > > > the
> > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > method may not necessarily result in lazy
> > > > > > > > initialization.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > perhaps
> > > > the
> > > > > > > > > encryption
> > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > should be implemented in the CatalogStore
> > > when
> > > > > the
> > > > > > > data
> > > > > > > > > is
> > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > saved. For instance, the FileCatalogStore
> > > could
> > > > > > > encrypt
> > > > > > > > > the
> > > > > > > > > > > > data
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > MemoryCatalogStore
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Do we really need the
> > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > MemoryCatalogStore
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > configurations
> > > > > > in
> > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > asynchronous
> > > > > > > loading
> > > > > > > > > of
> > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > would need to introduce additional cache.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > What if asynchronous registration
> failed?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This is also a critical concern. When
> > > > executing
> > > > > > DDL,
> > > > > > > > > DQL,
> > > > > > > > > > or
> > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > statements that reference a specific
> > Catalog,
> > > > the
> > > > > > > > Catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > will be initialized if it has not been
> > > > > initialized
> > > > > > > yet.
> > > > > > > > > If
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > initialization process fails, these
> > > statements
> > > > > will
> > > > > > > not
> > > > > > > > > be
> > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs
> and
> > > > > > > > CatalogStore
> > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > catalog
> > > > > name,
> > > > > > > but
> > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > This issue can be resolved by referring
> to
> > > the
> > > > > > > previous
> > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > key principle is that the Catalog that
> has
> > > been
> > > > > > most
> > > > > > > > > > recently
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > will be given priority for subsequent
> use.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun
> Ye
> > <
> > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > believe
> > > > this
> > > > > > > Flip
> > > > > > > > > > could
> > > > > > > > > > > be
> > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > manages a
> > > > lot
> > > > > > of
> > > > > > > > > > > catalogs,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > submission of tasks becomes slow because
> of
> > > > > > loading a
> > > > > > > > > > number
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> Flink
> > > SQL
> > > > > > > through
> > > > > > > > > > > regular
> > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > to avoid loading all catalogs to improve
> > the
> > > > > speed
> > > > > > of
> > > > > > > > > task
> > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > questions:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. When creating a catalog with CREATE
> > > > CATALOG,
> > > > > > the
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration method is used by default.
> > What
> > > if
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > failed? And how to implement asynchronous
> > > > > > > registration?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. I also have the same question with
> > > > Shammon’s
> > > > > > > first
> > > > > > > > > > > > comment.
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > CatalogStore
> > > > > > > > > catalogstore
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogManager all have the same catalog
> > > name,
> > > > > but
> > > > > > > > > > > correponding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > different catalog instances. For example:
> > > > > > catalog1->
> > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog in
> > the
> > > > > > > > > catalogstore.
> > > > > > > > > > > > Would
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > happen? Maybe we should define `catalogs`
> > and
> > > > > > > > > > `catalogstore`
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > ruanhang1993@gmail.com
> > > > > > > > >
> > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi everyone,
Thank you everyone for your valuable input. If there are no further
questions or concerns about the FLIP-295[1], I would like to start voting
tomorrow (6/14).
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Best,
Feng
On Mon, Jun 12, 2023 at 10:54 AM Feng Jin <ji...@gmail.com> wrote:
> hi, Hang
>
> Thanks for your reply, Make sense to me, I updated the FLIP.
>
>
> Best,
>
> Feng
>
>
>
> On Fri, Jun 9, 2023 at 3:34 PM Hang Ruan <ru...@gmail.com> wrote:
>
>> Hi, Feng.
>>
>> I reviewed this FLIP again and found some little places that may need to
>> optimize.
>>
>> 1. `CatalogDescriptor` may need a private constructor.
>> 2. The method `initCatalog` in `CatalogManager` is not introduced.
>>
>> Best,
>> Hang
>>
>> Feng Jin <ji...@gmail.com> 于2023年6月6日周二 21:17写道：
>>
>> > Hi Leonard,
>> >
>> > Thanks for your reply.
>> >
>> > > 1. a How to construct a CatalogDescriptor ?
>> >
>> > I think it would be helpful to add a method for constructing a
>> > CatalogDescriptor, as you mentioned in 1.c. I will update the
>> documentation
>> > later.
>> >
>> > > 1.b How to visit the fields ? Could we use Configuration instead of
>> > Map<String, String> ?
>> >
>> > I believe that the use of Map<String, String> options is only intended
>> for
>> > creating a catalog and not for accessing internal parameters.
>> >
>> > Since all of the relevant parameters for CREATE CATALOG are also stored
>> in
>> > Map<String, String> options, my understanding is that using Map<String,
>> > String> options should suffice.
>> >
>> > Here is the implementation of execute CREATE CATALOG statement.
>> > ```java
>> > private TableResultInternal createCatalog(CreateCatalogOperation
>> operation)
>> > {
>> > String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
>> > try {
>> > String catalogName = operation.getCatalogName();
>> > Map<String, String> properties = operation.getProperties();
>> >
>> > Catalog catalog =
>> > FactoryUtil.createCatalog(
>> > catalogName,
>> > properties,
>> > tableConfig,
>> > resourceManager.getUserClassLoader());
>> > catalogManager.registerCatalog(catalogName, catalog);
>> >
>> > return TableResultImpl.TABLE_RESULT_OK;
>> > } catch (CatalogException e) {
>> > throw new ValidationException(exMsg, e);
>> > }
>> > }
>> > ```
>> >
>> >
>> > > 2. Do we have plan to offer a default CatalogStore if user didn’t
>> config
>> > this?
>> >
>> > Yes, the in-memory catalogStore will be used as the default CatalogStore
>> > even if the user has not configured one
>> >
>> >
>> > Best,
>> > Feng
>> >
>> >
>> > On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
>> >
>> > > Hi, Feng
>> > >
>> > > Sorry for reply late, but I’ve some comments about the FLIP
>> > >
>> > >
>> > > 1. The introduced Public class CatalogDescriptor seems missed some
>> > > necessary component
>> > > a) How to construct a CatalogDescriptor ?
>> > > b) How to visit the fields ? Could we use Configuration instead of
>> > > Map<String, String> ?
>> > > c) Could we offer a built-in factory method to build a
>> > CatalogDescriptor
>> > > like
>> > > public static CatalogDescriptor of(String catalogName,
>> Configuration
>> > > configuration)
>> > >
>> > > 2. The FLIP said “By default, there are two built-in CatalogStores
>> > > available: the In-Memory CatalogStore and the File CatalogStore” ,
>> > > Do we have plan to offer a default CatalogStore if user didn’t config
>> > > this? IIUC, users can obtains the benefits from lazy catalog
>> > > initialization If
>> > > we have a default catalogstore even it is in-memory.
>> > >
>> > > Best,
>> > > Leonard
>> > >
>> > >
>> > >
>> > > > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
>> > > >
>> > > > Hi everyone,
>> > > >
>> > > > Thank you everyone for your valuable input. If there are no further
>> > > > questions or concerns about the FLIP[1], I would like to start
>> voting
>> > > > tomorrow (6/7).
>> > > >
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >
>> > > >
>> > > > Best,
>> > > > Feng
>> > > >
>> > > >
>> > > > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hi Samrat,
>> > > >>
>> > > >> Thanks for your advice.
>> > > >>
>> > > >>> 1. The createCatalog method does not mention any exceptions being
>> > > >> thrown.
>> > > >>
>> > > >> CreateCatalog will throw CatalogException like registerCatalog. As
>> > > >> CatalogException is a RuntimeException,
>> > > >> there is no explicit declaration of throwing Exceptions in
>> > > CatalogManager
>> > > >> and TableEnvironment.
>> > > >> To avoid misunderstandings, I have added the "throw
>> CatalogException"
>> > > flag
>> > > >> to the createCatalog method definition of CatalogStore.
>> > > >>
>> > > >>> 2. Could you please provide an exhaustive list of the supported
>> > kinds?
>> > > >>
>> > > >> Sure, the documentation now includes the configuration of the
>> > built-in
>> > > >> CatalogStore as well as how to configure a custom CatalogStore.
>> > > >>
>> > > >>
>> > > >> Best,
>> > > >> Feng
>> > > >>
>> > > >>
>> > > >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
>> > > wrote:
>> > > >>
>> > > >>> Hi Feng,
>> > > >>>
>> > > >>> Thank you for providing the proposal. I believe this feature will
>> be
>> > > >>> highly
>> > > >>> valuable.
>> > > >>>
>> > > >>> I have a couple of inquiries:
>> > > >>>
>> > > >>> 1. According to the documentation [1], the createCatalog method
>> does
>> > > not
>> > > >>> mention any exceptions being thrown. However, I would like to
>> confirm
>> > > if
>> > > >>> it
>> > > >>> is always true that there will be no failures in all scenarios.
>> > Please
>> > > let
>> > > >>> me know if there is any additional information I may have missed.
>> > > >>>
>> > > >>> 2. Regarding the registration process using the
>> > > `table.catalog-store.kind`
>> > > >>> configuration, could you please provide an exhaustive list of the
>> > > >>> supported
>> > > >>> kinds?
>> > > >>> It would be great to have a comprehensive understanding of the
>> > > options
>> > > >>> available.
>> > > >>>
>> > > >>> Bests,
>> > > >>> Samrat
>> > > >>>
>> > > >>> [1]
>> > > >>>
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >>>
>> > > >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com>
>> > wrote:
>> > > >>>
>> > > >>>> Hi Hang and Jark
>> > > >>>>
>> > > >>>> Thank you very much for your reply.
>> > > >>>>
>> > > >>>> @Jark
>> > > >>>>> 1. Could you move the CatalogStore registration API to the
>> "Public
>> > > >>>> Interface" section?
>> > > >>>>
>> > > >>>> Indeed, it is more reasonable.
>> > > >>>>
>> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
>> > > >>>>
>> > > >>>> Sure, the names that you provided are indeed more logical and
>> > easier
>> > > >>> to
>> > > >>>> understand.
>> > > >>>>
>> > > >>>>> 3. About the open/close method in CatalogStoreFactory.
>> > > >>>>
>> > > >>>> I also agree with the proposed requirement scenario and design. I
>> > have
>> > > >>>> already made modifications to the interface.
>> > > >>>>
>> > > >>>> @Hang
>> > > >>>>
>> > > >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
>> > > close
>> > > >>>> the resource.
>> > > >>>>
>> > > >>>> I have already added the missing method.
>> > > >>>>
>> > > >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> > > >>>>
>> > > >>>> Sorry for the misspelling, I updated this part.
>> > > >>>>
>> > > >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
>> > > methods
>> > > >>>> to init or close its resource
>> > > >>>>
>> > > >>>> This is indeed a scenario that needs to be considered, and I have
>> > > added
>> > > >>>> relevant methods accordingly.
>> > > >>>> Additionally, I have included some relevant description
>> information
>> > in
>> > > >>> the
>> > > >>>> documentation to help others understand
>> > > >>>> why the open/close methods need to be added.
>> > > >>>>
>> > > >>>>
>> > > >>>> Best,
>> > > >>>> Feng
>> > > >>>>
>> > > >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com>
>> wrote:
>> > > >>>>
>> > > >>>>> Hi Jing,
>> > > >>>>>
>> > > >>>>> Thank you for the update.
>> > > >>>>>
>> > > >>>>> 1. Could you move the CatalogStore registration API to the
>> "Public
>> > > >>>>> Interface" section?
>> > > >>>>> "Proposed Changes" is more like a place to describe the
>> > > implementation
>> > > >>>>> details.
>> > > >>>>>
>> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
>> > > >>> Besides,
>> > > >>>>> the config key
>> > > >>>>> name should be hierarchical[1]. Therefore, it may be better to
>> use:
>> > > >>>>> "table.catalog-store.kind"
>> > > >>>>> "table.catalog-store.file.path"
>> > > >>>>>
>> > > >>>>> 3. I think Hang's suggestions make sense.
>> > > >>>>>
>> > > >>>>> Others look good to me.
>> > > >>>>>
>> > > >>>>> Best,
>> > > >>>>> Jark
>> > > >>>>>
>> > > >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
>> > > >>> wrote:
>> > > >>>>>
>> > > >>>>>> Hi, Feng.
>> > > >>>>>>
>> > > >>>>>> Thanks for the update.
>> > > >>>>>> The current design looks good to me. I have some minor
>> comments.
>> > > >>>>>>
>> > > >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init
>> or
>> > > >>> close
>> > > >>>>> the
>> > > >>>>>> resource. For example, when storing the information in MySQL,
>> the
>> > > >>> store
>> > > >>>>>> needs to open and close the connections.
>> > > >>>>>>
>> > > >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> > > >>>>>>
>> > > >>>>>> 3. About the usage in the sql gateway.
>> > > >>>>>> Considering the usage in sql gateway, the sql gateway may
>> create a
>> > > >>>>>> CatalogStore for each session.
>> > > >>>>>> If we are using the MySqlCatalogStore, there would be so many
>> > > >>>>> connections.
>> > > >>>>>> How can we reuse the connection among these sessions?
>> > > >>>>>> I think sql gateway need to maintain a connection pool in
>> > > >>>>>> the CatalogStoreFactory and each session get its own connection
>> > from
>> > > >>>> the
>> > > >>>>>> pool when it is created.
>> > > >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close`
>> methods
>> > > >>> to
>> > > >>>>> init
>> > > >>>>>> or close its resource.
>> > > >>>>>> Or is there a better way?
>> > > >>>>>>
>> > > >>>>>> Best,
>> > > >>>>>> Hang
>> > > >>>>>>
>> > > >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>> > > >>>>>>
>> > > >>>>>>> Thanks Jingsong.
>> > > >>>>>>>
>> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
>> > > >>>>>>>
>> > > >>>>>>> +1 For this, I have already updated FLIP.
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> Best,
>> > > >>>>>>> Feng
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
>> > > >>> jingsonglee0@gmail.com>
>> > > >>>>>>> wrote:
>> > > >>>>>>>
>> > > >>>>>>>> Thanks Feng,
>> > > >>>>>>>>
>> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see
>> many
>> > > >>>>>>>> functions are converted to createxxx from registerxxx.
>> > > >>>>>>>>
>> > > >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <
>> jinfeng1995@gmail.com
>> > > >>>>
>> > > >>>>>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>> Hi jark, thanks for your suggestion.
>> > > >>>>>>>>>
>> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>> > > >>> the
>> > > >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
>> > > >>>>> Otherwise,
>> > > >>>>>>>> there
>> > > >>>>>>>>> might be some data inconsistencies when CatalogStore is
>> > > >>> changed.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
>> > > >>>>>> TableEnv.
>> > > >>>>>>>>> Therefore, my current proposal is to add a method to
>> configure
>> > > >>>> the
>> > > >>>>>>>>> CatalogStore in EnvironmentSettings.
>> > > >>>>>>>>>
>> > > >>>>>>>>> // Initialize a catalog Store
>> > > >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
>> > > >>>>>>>>>
>> > > >>>>>>>>> // set up the Table API
>> > > >>>>>>>>> final EnvironmentSettings settings =
>> > > >>>>>>>>>
>> > > >>> EnvironmentSettings.newInstance().inBatchMode()
>> > > >>>>>>>>> .withCatalogStore(catalogStore)
>> > > >>>>>>>>> .build();
>> > > >>>>>>>>>
>> > > >>>>>>>>> final TableEnvironment tableEnv =
>> > > >>>>>>>> TableEnvironment.create(settings);
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>> > > >>>> default
>> > > >>>>>>>> method,
>> > > >>>>>>>>> not an interface method?
>> > > >>>>>>>>>
>> > > >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
>> > > >>> that
>> > > >>>> as
>> > > >>>>> a
>> > > >>>>>>> new
>> > > >>>>>>>>> interface, the CatalogStoreFactory should not have a default
>> > > >>>> method
>> > > >>>>>> but
>> > > >>>>>>>> an
>> > > >>>>>>>>> interface method. I have already modified the interface.
>> > > >>>>>>>>>
>> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>> > > >>>>>>>>> deprecated`registerCatalog`.
>> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
>> > > >>> mention
>> > > >>>> the
>> > > >>>>>>>> changed
>> > > >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
>> > > >>>> other
>> > > >>>>>>>>> persisted catalog store) is used.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> [1].
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> Best,
>> > > >>>>>>>>> Feng
>> > > >>>>>>>>>
>> > > >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
>> > > >>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
>> > > >>>>>>>>>> The current design looks pretty good to me. I just have
>> some
>> > > >>>>> minor
>> > > >>>>>>>>>> comments.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>> > > >>> the
>> > > >>>>>>>> CatalogStore
>> > > >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
>> > > >>> there
>> > > >>>>>> might
>> > > >>>>>>> be
>> > > >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>> > > >>>> default
>> > > >>>>>>>> method,
>> > > >>>>>>>>>> not an interface method?
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>> > > >>>>> deprecated
>> > > >>>>>>>>>> `registerCatalog`.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
>> > > >>> mention
>> > > >>>> the
>> > > >>>>>>>> changed
>> > > >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore
>> (or
>> > > >>>>> other
>> > > >>>>>>>>>> persisted catalog store) is used.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Best,
>> > > >>>>>>>>>> Jark
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
>> > > >>> jinfeng1995@gmail.com>
>> > > >>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> @Ron
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
>> > > >>>> method,
>> > > >>>>>> do
>> > > >>>>>>> we
>> > > >>>>>>>>>> need
>> > > >>>>>>>>>>> to provide a default implementation?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Yes, we will provide a default
>> > > >>> InMemoryCatalogStoreFactory to
>> > > >>>>>>> create
>> > > >>>>>>>> an
>> > > >>>>>>>>>>> InMemoryCatalogStore.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
>> > > >>> initializing
>> > > >>>>> it,
>> > > >>>>>>>> whether
>> > > >>>>>>>>>> we
>> > > >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Yes, in the current design, catalogs are stored as
>> > > >>> snapshots,
>> > > >>>>> and
>> > > >>>>>>>> once
>> > > >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
>> > > >>>>>> Catalog>
>> > > >>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
>> > > >>> the
>> > > >>>>>> primary
>> > > >>>>>>>>>> source
>> > > >>>>>>>>>>> for obtaining the corresponding Catalog.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
>> > > >>>>>>>>>>> `catalog.store.path`?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
>> > > >>> seem a
>> > > >>>>> bit
>> > > >>>>>>>>>> strange. I
>> > > >>>>>>>>>>> will update the FLIP.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Thank you for the review. I have made the necessary
>> > > >>>>> corrections.
>> > > >>>>>>>>>>> Regarding the modifications made to the Public Interface,
>> > > >>> I
>> > > >>>>> have
>> > > >>>>>>> also
>> > > >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Best,
>> > > >>>>>>>>>>> Feng
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
>> > > >>>> zjureel@gmail.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> Hi feng,
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Thanks for updating, I have some minor comments
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
>> > > >>>>>> `Public
>> > > >>>>>>>>>>>> Interfaces`, it is not a public interface.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
>> > > >>> and
>> > > >>>>>>>>>>>> `CatalogStoreFactory`
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
>> > > >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
>> > > >>>>> should
>> > > >>>>>> be
>> > > >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
>> > > >>>>>>>>>>>> catalogStore.get(catalogName);`
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
>> > > >>>> ron9.liu@gmail.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Hi, Feng
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
>> > > >>>> useful
>> > > >>>>>> for
>> > > >>>>>>>>>> catalog
>> > > >>>>>>>>>>>>> management.
>> > > >>>>>>>>>>>>> I have some small questions:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> 1. Regarding the
>> > > >>> CatalogStoreFactory#createCatalogStore
>> > > >>>>>> method,
>> > > >>>>>>>> do we
>> > > >>>>>>>>>>>> need
>> > > >>>>>>>>>>>>> to provide a default implementation?
>> > > >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
>> > > >>>> initializing
>> > > >>>>>> it,
>> > > >>>>>>>>>> whether
>> > > >>>>>>>>>>> we
>> > > >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
>> > > >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
>> > > >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
>> > > >>> them to
>> > > >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>> Ron
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
>> > > >>> 21:19写道：
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Hi yuxia
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
>> > > >>>>> register
>> > > >>>>>>> the
>> > > >>>>>>>>>>> Catalog,
>> > > >>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>> initialize it and open it. right?
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
>> > > >>>>>> original
>> > > >>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
>> > > >>>>>>>>>> registerCatalog
>> > > >>>>>>>>>>> so
>> > > >>>>>>>>>>>>>> that parameter validation can be performed.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
>> > > >>>>>>> reflected
>> > > >>>>>>>> in
>> > > >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
>> > > >>>>> catalog
>> > > >>>>>>>>>>>>> configurations,
>> > > >>>>>>>>>>>>>> only initialization is required in getCatalog.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
>> > > >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Hi, Feng.
>> > > >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
>> > > >>>>>>>> initialization*.
>> > > >>>>>>>>>>> If
>> > > >>>>>>>>>>>>> i'm
>> > > >>>>>>>>>>>>>>> wrong, please correct me.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
>> > > >>>> access
>> > > >>>>>> the
>> > > >>>>>>>>>>> catalog,
>> > > >>>>>>>>>>>>> then
>> > > >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
>> > > >>>>> Changes,
>> > > >>>>>>>> once we
>> > > >>>>>>>>>>>>>> register
>> > > >>>>>>>>>>>>>>> the Catalog,
>> > > >>>>>>>>>>>>>>> we initialize it and open it. right?
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>> Yuxia
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> ----- 原始邮件 -----
>> > > >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>> > > >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
>> > > >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
>> > > >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
>> > > >>>>>> Catalog
>> > > >>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>> and asynchronous registration
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> One of the major changes is that current design
>> > > >>> will
>> > > >>>>>>> provide
>> > > >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
>> > > >>>> of a
>> > > >>>>>>>> cache,
>> > > >>>>>>>>>>> which
>> > > >>>>>>>>>>>>>> means
>> > > >>>>>>>>>>>>>>> once it has been initialized, any changes done by
>> > > >>>> other
>> > > >>>>>>>> sessions
>> > > >>>>>>>>>>> will
>> > > >>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
>> > > >>>>>> further
>> > > >>>>>>>>>>>> improvement.
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
>> > > >>>>>>>> jinfeng1995@gmail.com>
>> > > >>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
>> > > >>>>>> progress
>> > > >>>>>>>> of
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
>> > > >>> Shammon
>> > > >>>> FY,
>> > > >>>>>>>> ShengKai
>> > > >>>>>>>>>>> Fang
>> > > >>>>>>>>>>>>>> and I
>> > > >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
>> > > >>>>>> solution
>> > > >>>>>>>> for
>> > > >>>>>>>>>>> Flink
>> > > >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
>> > > >>>> have
>> > > >>>>>>>> updated
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> final
>> > > >>>>>>>>>>>>>>>> solution in FLIP.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> 1.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
>> > > >>>>>>>> configuration
>> > > >>>>>>>>>>>> similar
>> > > >>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>> TableDescriptor.
>> > > >>>>>>>>>>>>>>>> 2.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
>> > > >>>>>>>>>>> storeCatalog(String
>> > > >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
>> > > >>>>>> CatalogDescriptor
>> > > >>>>>>>>>>>>>>> getCatalog(String)
>> > > >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
>> > > >>>> Catalog
>> > > >>>>>>>> instance.
>> > > >>>>>>>>>>>> This
>> > > >>>>>>>>>>>>>> way,
>> > > >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
>> > > >>> saving
>> > > >>>>> and
>> > > >>>>>>>>>>> retrieving
>> > > >>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> configurations without having to initialize
>> > > >>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>> 3.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> The default registerCatalog(String
>> > > >>> catalogName,
>> > > >>>>>>> Catalog
>> > > >>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
>> > > >>>>>>> deprecated.
>> > > >>>>>>>>>>>>>>>> 4.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> A new function registerCatalog(String
>> > > >>>> catalogName,
>> > > >>>>>>>>>>>>> CatalogDescriptor
>> > > >>>>>>>>>>>>>>>> catalog) will be added to serve as the
>> > > >>> default
>> > > >>>>>>>> registration
>> > > >>>>>>>>>>>>> function
>> > > >>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>> catalogs in CatalogManager.
>> > > >>>>>>>>>>>>>>>> 5.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
>> > > >>>> will
>> > > >>>>>>>> remain
>> > > >>>>>>>>>>>>> unchanged
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>> save initialized catalogs.This means that
>> > > >>>> deletion
>> > > >>>>>>>>>> operations
>> > > >>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>> one
>> > > >>>>>>>>>>>>>>>> session won't synchronize with other
>> > > >>> sessions.
>> > > >>>>>>>>>>>>>>>> 6.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> To support multi-session synchronization
>> > > >>>> scenarios
>> > > >>>>>> for
>> > > >>>>>>>>>>> deletions
>> > > >>>>>>>>>>>>>> later
>> > > >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
>> > > >>>>>>>>>>> configurable.There
>> > > >>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>> three possible situations:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
>> > > >>>>>>>> automatically
>> > > >>>>>>>>>>> clear
>> > > >>>>>>>>>>>>>>>> long-unused catalogs.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
>> > > >>>>>> getCatalog
>> > > >>>>>>>>>>> creates a
>> > > >>>>>>>>>>>>> new
>> > > >>>>>>>>>>>>>>>> instance.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> This is the document for discussion:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> This is the final proposal document:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Thank you very much for your attention and
>> > > >>>>> suggestions
>> > > >>>>>> on
>> > > >>>>>>>> this
>> > > >>>>>>>>>>>>> FLIP. A
>> > > >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
>> > > >>> on
>> > > >>>>> the
>> > > >>>>>>>> entire
>> > > >>>>>>>>>>>> design
>> > > >>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>> organizing offline discussions.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
>> > > >>>> about
>> > > >>>>>>> this
>> > > >>>>>>>> FLIP
>> > > >>>>>>>>>>>>> please
>> > > >>>>>>>>>>>>>>> feel
>> > > >>>>>>>>>>>>>>>> free to share.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
>> > > >>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
>> > > >>> to
>> > > >>>> me.
>> > > >>>>>> We
>> > > >>>>>>>> could
>> > > >>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
>> > > >>>> common
>> > > >>>>>>>> built-in
>> > > >>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
>> > > >>> every
>> > > >>>>>>>>>> CatalogSotre
>> > > >>>>>>>>>>>>>>>>> implementation has to take care of it.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
>> > > >>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Hi Jing,
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Thanks for your reply.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>> > > >>>>> suggest
>> > > >>>>>>> you
>> > > >>>>>>>>>>>>> completely
>> > > >>>>>>>>>>>>>>> walk
>> > > >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
>> > > >>> the
>> > > >>>>> same
>> > > >>>>>>>> time, I
>> > > >>>>>>>>>>>>> greatly
>> > > >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
>> > > >>>> thoroughly
>> > > >>>>>>>> checked
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>> entire
>> > > >>>>>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>>> and made corrections to these issues.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
>> > > >>> FLIP
>> > > >>>>>>> design,
>> > > >>>>>>>>>>>>>>> CatalogManager
>> > > >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
>> > > >>> being
>> > > >>>>>>>> configured.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
>> > > >>> was
>> > > >>>> not
>> > > >>>>>>>> necessary
>> > > >>>>>>>>>>>>> because
>> > > >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
>> > > >>>> catalogs
>> > > >>>>>> to
>> > > >>>>>>>> store
>> > > >>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>> instances.
>> > > >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
>> > > >>>>>>> Therefore, I
>> > > >>>>>>>>>>>> modified
>> > > >>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>> part
>> > > >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
>> > > >>> Catalog>
>> > > >>>>>>> catalogs
>> > > >>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>> CatalogManager.
>> > > >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
>> > > >>> serve
>> > > >>>> as
>> > > >>>>>> the
>> > > >>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
>> > > >>>>>>>> functionality of
>> > > >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
>> > > >>>>>>>>>>>>>>>>>> The previous plan that kept
>> > > >>>>>>>> Map<String,Catalog>catalogs has
>> > > >>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>> moved
>> > > >>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>> Rejected Alternatives.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
>> > > >>>>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There are still many places contain
>> > > >>>>> inconsistent
>> > > >>>>>>>> content,
>> > > >>>>>>>>>>>> e.g.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
>> > > >>> used.
>> > > >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
>> > > >>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
>> > > >>> not
>> > > >>>>> tell
>> > > >>>>>>>> what the
>> > > >>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>> doing.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>> > > >>>>> suggest
>> > > >>>>>>> you
>> > > >>>>>>>>>>>> completely
>> > > >>>>>>>>>>>>>>> walk
>> > > >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
>> > > >>> issues.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
>> > > >>> mean
>> > > >>>>> that
>> > > >>>>>>> you
>> > > >>>>>>>> will
>> > > >>>>>>>>>>>> build
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
>> > > >>> is a
>> > > >>>>> very
>> > > >>>>>>>>>> different
>> > > >>>>>>>>>>>>>> design
>> > > >>>>>>>>>>>>>>>>>> concept
>> > > >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
>> > > >>> I am
>> > > >>>>> not
>> > > >>>>>>>>>> mistaken,
>> > > >>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
>> > > >>>> work
>> > > >>>>>>>> without
>> > > >>>>>>>>>>>> Optional
>> > > >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
>> > > >>>>> reason
>> > > >>>>>>>> why I
>> > > >>>>>>>>>>>>> mentioned
>> > > >>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
>> > > >>>>> Optional
>> > > >>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>> correct.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
>> > > >>>>>>>>>>>>> jinfeng1995@gmail.com>
>> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Hi Jing
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
>> > > >>>> FLIP
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
>> > > >>>> review. I
>> > > >>>>>>> have
>> > > >>>>>>>>>>> already
>> > > >>>>>>>>>>>>> made
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> relevant changes.
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
>> > > >>> the
>> > > >>>>>>>>>> multi-instance
>> > > >>>>>>>>>>>>>>>> scenarios, I
>> > > >>>>>>>>>>>>>>>>>> am
>> > > >>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
>> > > >>> multiple
>> > > >>>>>>>> scenarios
>> > > >>>>>>>>>> with
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> current
>> > > >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
>> > > >>>>> remove
>> > > >>>>>>>>>>> 'Map<String,
>> > > >>>>>>>>>>>>>>>> Catalog>'
>> > > >>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
>> > > >>>>>>>>>> InMemoryCatalogStore.
>> > > >>>>>>>>>>>> This
>> > > >>>>>>>>>>>>>>> way,
>> > > >>>>>>>>>>>>>>>>>> users
>> > > >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
>> > > >>>>> achieve
>> > > >>>>>>>>>>>>> multi-instance
>> > > >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
>> > > >>>> think?
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
>> > > >>>>>>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
>> > > >>> are
>> > > >>>>>> still
>> > > >>>>>>>> some
>> > > >>>>>>>>>>> NIT
>> > > >>>>>>>>>>>>>> issues
>> > > >>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>>>>>> like:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
>> > > >>>> has
>> > > >>>>>> been
>> > > >>>>>>>> used
>> > > >>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
>> > > >>> example.
>> > > >>>> It
>> > > >>>>>>>> should be
>> > > >>>>>>>>>>>> fine
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> use
>> > > >>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
>> > > >>> after
>> > > >>>> you
>> > > >>>>>>>> submit
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>> PR.
>> > > >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
>> > > >>>> somewhere
>> > > >>>>> in
>> > > >>>>>>> the
>> > > >>>>>>>>>>>> rejected
>> > > >>>>>>>>>>>>>>>> section
>> > > >>>>>>>>>>>>>>>>>>> which
>> > > >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
>> > > >>> it
>> > > >>>>>>>> consistent.
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
>> > > >>>>>>>> multi-instance
>> > > >>>>>>>>>>>>>>> scenarios,
>> > > >>>>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>>> am
>> > > >>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
>> > > >>> behaviour.
>> > > >>>> If
>> > > >>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
>> > > >>>> once
>> > > >>>>>> the
>> > > >>>>>>>>>> related
>> > > >>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> has
>> > > >>>>>>>>>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
>> > > >>> another
>> > > >>>>>>>> instance.
>> > > >>>>>>>>>> Did
>> > > >>>>>>>>>>> I
>> > > >>>>>>>>>>>>> miss
>> > > >>>>>>>>>>>>>>>>>>> something?
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
>> > > >>> Jin
>> > > >>>> <
>> > > >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> @Jing
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
>> > > >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
>> > > >>> name, I
>> > > >>>>>> have
>> > > >>>>>>>>>> updated
>> > > >>>>>>>>>>> it
>> > > >>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>> documentation.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
>> > > >>> should
>> > > >>>> be
>> > > >>>>>>>>>> implemented
>> > > >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
>> > > >>>>>> alternative
>> > > >>>>>>> 1
>> > > >>>>>>>>>> turns
>> > > >>>>>>>>>>>> out
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
>> > > >>> this
>> > > >>>>> part
>> > > >>>>>>> of
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> content.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
>> > > >>>> forward,
>> > > >>>>>> the
>> > > >>>>>>>>>> option 2
>> > > >>>>>>>>>>>>>>> solution
>> > > >>>>>>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
>> > > >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
>> > > >>>>> offline,
>> > > >>>>>>> we
>> > > >>>>>>>>>> think
>> > > >>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>> solution
>> > > >>>>>>>>>>>>>>>>>> 2
>> > > >>>>>>>>>>>>>>>>>>>>> might
>> > > >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
>> > > >>>>>>>> inconsistency
>> > > >>>>>>>>>>>> issues.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
>> > > >>>>> descriptions
>> > > >>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> content.
>> > > >>>>>>>>>>>>>>>> Would
>> > > >>>>>>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>>>>> to clean them up?
>> > > >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
>> > > >>>> document
>> > > >>>>>> and
>> > > >>>>>>>>>>>> appreciate
>> > > >>>>>>>>>>>>>> your
>> > > >>>>>>>>>>>>>>>>>>>>> suggestions.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
>> > > >>> in
>> > > >>>>>>>> `Rejected
>> > > >>>>>>>>>>>>>>> Alternatives`
>> > > >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
>> > > >>> `Rejected
>> > > >>>>>>>>>> Alternatives`.
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> Best
>> > > >>>>>>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
>> > > >>> Shammon
>> > > >>>>> FY
>> > > >>>>>> <
>> > > >>>>>>>>>>>>>>> zjureel@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
>> > > >>>>>> `Proposed
>> > > >>>>>>>>>>> Changes`,
>> > > >>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>>> put
>> > > >>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
>> > > >>>>>>>> Alternatives`? I
>> > > >>>>>>>>>>> think
>> > > >>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>>>>> help
>> > > >>>>>>>>>>>>>>>>>>>> us
>> > > >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
>> > > >>> Jing
>> > > >>>> Ge
>> > > >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
>> > > >>> am
>> > > >>>>>> still
>> > > >>>>>>>>>> confused
>> > > >>>>>>>>>>>>> after
>> > > >>>>>>>>>>>>>>>>>>> completely
>> > > >>>>>>>>>>>>>>>>>>>>>>> reading
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
>> > > >>>> questions:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
>> > > >>>>> registerCatalog()
>> > > >>>>>>> and
>> > > >>>>>>>>>>>>>> addCatalog()
>> > > >>>>>>>>>>>>>>>> have
>> > > >>>>>>>>>>>>>>>>>> no
>> > > >>>>>>>>>>>>>>>>>>>> big
>> > > >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
>> > > >>>> One
>> > > >>>>> of
>> > > >>>>>>>> them is
>> > > >>>>>>>>>>>>>>> responsible
>> > > >>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>> data
>> > > >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
>> > > >>>>>> persistCatalog()?
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
>> > > >>>> Map<String,
>> > > >>>>>>>> Catalog>
>> > > >>>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>> used
>> > > >>>>>>>>>>>>>>>>>>> as a
>> > > >>>>>>>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
>> > > >>> data
>> > > >>>>>>>> persistence.
>> > > >>>>>>>>>> I
>> > > >>>>>>>>>>>>> would
>> > > >>>>>>>>>>>>>>>>> suggest
>> > > >>>>>>>>>>>>>>>>>>>>>> describing
>> > > >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
>> > > >>>>> clearly
>> > > >>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>> Some
>> > > >>>>>>>>>>>>>>>>> common
>> > > >>>>>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
>> > > >>>> i.e.
>> > > >>>>>> data
>> > > >>>>>>>> in
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> store
>> > > >>>>>>>>>>>>>>>>>>>>>>> should
>> > > >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
>> > > >>>> instance
>> > > >>>>>>>> should be
>> > > >>>>>>>>>>>> found
>> > > >>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> store
>> > > >>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
>> > > >>>>>> initialized
>> > > >>>>>>>> or it
>> > > >>>>>>>>>>> will
>> > > >>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>> lazy
>> > > >>>>>>>>>>>>>>>>>>>>>> initialized)
>> > > >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
>> > > >>>>>>> consistency
>> > > >>>>>>>> will
>> > > >>>>>>>>>>> be
>> > > >>>>>>>>>>>>>> taken
>> > > >>>>>>>>>>>>>>>> care
>> > > >>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>> while
>> > > >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
>> > > >>>>>> forward,
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>> option
>> > > >>>>>>>>>>>>> 2
>> > > >>>>>>>>>>>>>>>>> solution
>> > > >>>>>>>>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
>> > > >>>>>> because,
>> > > >>>>>>>> afaiu,
>> > > >>>>>>>>>>>>> issues
>> > > >>>>>>>>>>>>>>>>>> mentioned
>> > > >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
>> > > >>>>> solved
>> > > >>>>>>>> yet. Do
>> > > >>>>>>>>>>> you
>> > > >>>>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>>>> want
>> > > >>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>> propose
>> > > >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
>> > > >>>> suggestions
>> > > >>>>>> for
>> > > >>>>>>>> both
>> > > >>>>>>>>>> of
>> > > >>>>>>>>>>>>> them?
>> > > >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
>> > > >>>> there
>> > > >>>>>> are
>> > > >>>>>>>> some
>> > > >>>>>>>>>>>>>>> inconsistent
>> > > >>>>>>>>>>>>>>>>>>>>> descriptions
>> > > >>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
>> > > >>>> clean
>> > > >>>>>>> them
>> > > >>>>>>>> up?
>> > > >>>>>>>>>>>>> Thanks!
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Jing
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
>> > > >>> Feng
>> > > >>>>>> Jin <
>> > > >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
>> > > >>> and
>> > > >>>> I
>> > > >>>>>>>> completely
>> > > >>>>>>>>>>>> agree
>> > > >>>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>> your
>> > > >>>>>>>>>>>>>>>>>>>> point
>> > > >>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> view.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
>> > > >>>>>> complicated
>> > > >>>>>>>> the
>> > > >>>>>>>>>>> whole
>> > > >>>>>>>>>>>>>> issue.
>> > > >>>>>>>>>>>>>>>>> First
>> > > >>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>> foremost,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
>> > > >>>> persistence
>> > > >>>>>> of
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> Catalog's
>> > > >>>>>>>>>>>>>>>>>>>> Configuration.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
>> > > >>>>>> persistence
>> > > >>>>>>>> for
>> > > >>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>> Configuration,
>> > > >>>>>>>>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
>> > > >>> to
>> > > >>>> the
>> > > >>>>>>>> Catalog
>> > > >>>>>>>>>>>>>> interface.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
>> > > >>> Catalog
>> > > >>>>>>>> instance
>> > > >>>>>>>>>> to a
>> > > >>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>> String>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
>> > > >>>>>>>> implementation
>> > > >>>>>>>>>> will
>> > > >>>>>>>>>>>>> throw
>> > > >>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>> exception.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> /**
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
>> > > >>> the
>> > > >>>>>>>> properties
>> > > >>>>>>>>>> of
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>> object.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> *
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
>> > > >>> the
>> > > >>>>>>>> properties
>> > > >>>>>>>>>> of
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>> object
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
>> > > >>>>>>>> UnsupportedOperationException if
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> implementing
>> > > >>>>>>>>>>>>>>>>>>>>> class
>> > > >>>>>>>>>>>>>>>>>>>>>>> does
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> not override
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> * the default
>> > > >>> implementation
>> > > >>>> of
>> > > >>>>>> this
>> > > >>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> */
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
>> > > >>>>>>>> toProperties()
>> > > >>>>>>>>>> {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> throw new
>> > > >>>>>>>>>>>> UnsupportedOperationException("Please
>> > > >>>>>>>>>>>>>>>>> implement
>> > > >>>>>>>>>>>>>>>>>>>>>>>> toProperties
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
>> > > >>>> follows:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
>> > > >>> configured a
>> > > >>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>> toConfiguration()
>> > > >>>>>>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
>> > > >>> registering a
>> > > >>>>>>> Catalog
>> > > >>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>> > > >>>> catalogName,
>> > > >>>>>>>> Catalog
>> > > >>>>>>>>>>>>> catalog).
>> > > >>>>>>>>>>>>>>> The
>> > > >>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
>> > > >>>> Map<String,
>> > > >>>>>>>> String>
>> > > >>>>>>>>>>>>> properties
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>> saved
>> > > >>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
>> > > >>>>>> instances
>> > > >>>>>>>> do
>> > > >>>>>>>>>> not
>> > > >>>>>>>>>>>>>>> implement
>> > > >>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>> method,
>> > > >>>>>>>>>>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
>> > > >>>>>>>> CatalogStore,
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> toConfiguration()
>> > > >>>>>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
>> > > >>>>>>> consistency
>> > > >>>>>>>> with
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> original
>> > > >>>>>>>>>>>>>>>>>>>> process.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
>> > > >>>>>> Catalog>
>> > > >>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>> at
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
>> > > >>>>>> conflicts
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
>> > > >>>>> itself
>> > > >>>>>>> and
>> > > >>>>>>>>>>> provide
>> > > >>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>>> dedicated
>> > > >>>>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
>> > > >>> loading,
>> > > >>>>> such
>> > > >>>>>>> as
>> > > >>>>>>>>>>>>>>>>> ConfigurationCatalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> public class
>> > > >>> ConfigurationCatalog
>> > > >>>>>>>> implements
>> > > >>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>> {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>> ConfigurationCatalog(Map<String,
>> > > >>>>>>> String>
>> > > >>>>>>>>>>>>> properties)
>> > > >>>>>>>>>>>>>> {
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
>> > > >>> the
>> > > >>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
>> > > >>> 10:03 AM
>> > > >>>>>> Shammon
>> > > >>>>>>>> FY <
>> > > >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
>> > > >>> the
>> > > >>>>>>>> functionality
>> > > >>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>> `CatalogStore`,
>> > > >>>>>>>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
>> > > >>>> multiple
>> > > >>>>>>>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>> > > >>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
>> > > >>>>>> implementation
>> > > >>>>>>> in
>> > > >>>>>>>>>> Trino
>> > > >>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>> also
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> same,
>> > > >>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
>> > > >>> only
>> > > >>>>> has
>> > > >>>>>>> one
>> > > >>>>>>>>>>>>>>> `createCatalog`,
>> > > >>>>>>>>>>>>>>>>>> which
>> > > >>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>> save
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
>> > > >>>>>> `CatalogStore`.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
>> > > >>>>>> another
>> > > >>>>>>>>>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
>> > > >>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
>> > > >>> `Map<String,
>> > > >>>>>>> Catalog>
>> > > >>>>>>>>>>>> catalogs`
>> > > >>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>> `CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
>> > > >>>> mentioned
>> > > >>>>>>>> before,
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> `Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
>> > > >>>>>>>> `CatalogManager`.
>> > > >>>>>>>>>> How
>> > > >>>>>>>>>>>>> about
>> > > >>>>>>>>>>>>>>>>> saving
>> > > >>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
>> > > >>> catalogs`
>> > > >>>>> and
>> > > >>>>>>>>>>>> `CatalogStore
>> > > >>>>>>>>>>>>>>>>>>> catalogStore`
>> > > >>>>>>>>>>>>>>>>>>>>>>>> together
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
>> > > >>> added
>> > > >>>> in
>> > > >>>>>>>>>>>>> `CatalogManager`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>> > > >>> 5:22 PM
>> > > >>>>> Feng
>> > > >>>>>>>> Jin <
>> > > >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
>> > > >>> I
>> > > >>>> am
>> > > >>>>>> very
>> > > >>>>>>>> sorry
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>> misunderstanding
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> caused
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
>> > > >>>>> original
>> > > >>>>>>>>>>> discussion.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>> > > >>>>>> pre-discussion
>> > > >>>>>>>> [1]
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
>> > > >>> such a
>> > > >>>>>>>> discussion
>> > > >>>>>>>>>>>> before.
>> > > >>>>>>>>>>>>>>>>> However,
>> > > >>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> designing
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
>> > > >>> that
>> > > >>>>> the
>> > > >>>>>>>> logic of
>> > > >>>>>>>>>>>>>>>>> CatalogManager
>> > > >>>>>>>>>>>>>>>>>>>> itself
>> > > >>>>>>>>>>>>>>>>>>>>>>>> doesn't
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
>> > > >>>> cannot
>> > > >>>>>>> only
>> > > >>>>>>>>>>> persist
>> > > >>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>> instances
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
>> > > >>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> might
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
>> > > >>>> Catalogs,
>> > > >>>>>>>> because
>> > > >>>>>>>>>> in
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>> end
>> > > >>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>>>>>>>> need
>> > > >>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> save
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
>> > > >>>>> corresponding
>> > > >>>>>> to
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>> instances.
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
>> > > >>>>>>> CatalogStore
>> > > >>>>>>>>>>>> interface
>> > > >>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
>> > > >>> this
>> > > >>>>>> part, I
>> > > >>>>>>>> also
>> > > >>>>>>>>>>> took
>> > > >>>>>>>>>>>>>>>>> inspiration
>> > > >>>>>>>>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Trino's
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>> > > >>>> handling
>> > > >>>>>>>> catalog
>> > > >>>>>>>>>> with
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> same
>> > > >>>>>>>>>>>>>>>>> name
>> > > >>>>>>>>>>>>>>>>>>>>> looks a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> little
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>> > > >>>> suggestion. I
>> > > >>>>>>> will
>> > > >>>>>>>>>>> provide
>> > > >>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>> detailed
>> > > >>>>>>>>>>>>>>>>>>>>>> description
>> > > >>>>>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
>> > > >>> part,
>> > > >>>>> and
>> > > >>>>>>> add
>> > > >>>>>>>> it
>> > > >>>>>>>>>> to
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
>> > > >>>>>>>> `addCatalog`
>> > > >>>>>>>>>>> still
>> > > >>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>> confused
>> > > >>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>> me.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
>> > > >>>>>> catalog
>> > > >>>>>>> to
>> > > >>>>>>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>>>>>> how
>> > > >>>>>>>>>>>>>>>>>>> about
>> > > >>>>>>>>>>>>>>>>>>>>>>>> renaming
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
>> > > >>> very
>> > > >>>>>>> personal
>> > > >>>>>>>>>>>> opinion,
>> > > >>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>> decide
>> > > >>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> take
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
>> > > >>>>> intuitive
>> > > >>>>>>> to
>> > > >>>>>>>> me.
>> > > >>>>>>>>>> I
>> > > >>>>>>>>>>>>> don't
>> > > >>>>>>>>>>>>>>> see
>> > > >>>>>>>>>>>>>>>>> any
>> > > >>>>>>>>>>>>>>>>>>>>> problem
>> > > >>>>>>>>>>>>>>>>>>>>>>> with
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
>> > > >>>>> statement,
>> > > >>>>>>>> which
>> > > >>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>> called?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>> > > >>>>>> `addCatalog`?
>> > > >>>>>>>> I'm
>> > > >>>>>>>>>>>>> wondering
>> > > >>>>>>>>>>>>>>>>> whether
>> > > >>>>>>>>>>>>>>>>>>> user
>> > > >>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>> add a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
>> > > >>>>>> stement.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
>> > > >>>> original
>> > > >>>>>>>> design
>> > > >>>>>>>>>> was
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>> add
>> > > >>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>> directly
>> > > >>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
>> > > >>> would
>> > > >>>>>>> disrupt
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> existing
>> > > >>>>>>>>>>>>>>>>> logic.
>> > > >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> think
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
>> > > >>>>>>>> configuration to
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> initialize
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
>> > > >>> the
>> > > >>>>> same
>> > > >>>>>>> time
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
>> > > >>> neccessary
>> > > >>>> to
>> > > >>>>>>>> provide a
>> > > >>>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>>>>> implmentation
>> > > >>>>>>>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>> > > >>>>> `CatalogStoreFactory`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
>> > > >>>>>> otherwise
>> > > >>>>>>> we
>> > > >>>>>>>>>> would
>> > > >>>>>>>>>>>>> need
>> > > >>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>> introduce
>> > > >>>>>>>>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> additional
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
>> > > >>>> configuration
>> > > >>>>>> for
>> > > >>>>>>>> lazy
>> > > >>>>>>>>>>>>> loading.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>> > > >>>>>>> registration
>> > > >>>>>>>> for
>> > > >>>>>>>>>>>>> catalog.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
>> > > >>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
>> > > >>>>>> interface
>> > > >>>>>>>>>> because
>> > > >>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>>>>>> an
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
>> > > >>>>>>>> initialization
>> > > >>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>> than
>> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
>> > > >>>>>>>> inaccurate. It
>> > > >>>>>>>>>>>>> should
>> > > >>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>> lazy
>> > > >>>>>>>>>>>>>>>>>>>>>>> registration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
>> > > >>>>>>>> registration. I
>> > > >>>>>>>>>>> have
>> > > >>>>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>>>> updated
>> > > >>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> title
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > >
>> >
>> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>> > > >>>> 4:27 PM
>> > > >>>>>>>> Shammon
>> > > >>>>>>>>>> FY <
>> > > >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
>> > > >>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
>> > > >>>>> method
>> > > >>>>>>>> named
>> > > >>>>>>>>>>>>>> `addCatalog`
>> > > >>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
>> > > >>> the
>> > > >>>>>> memory
>> > > >>>>>>>>>> catalog
>> > > >>>>>>>>>>>> is a
>> > > >>>>>>>>>>>>>>>>> `cache`,
>> > > >>>>>>>>>>>>>>>>>> I
>> > > >>>>>>>>>>>>>>>>>>>>> think
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
>> > > >>>>>> exposed
>> > > >>>>>>> to
>> > > >>>>>>>>>>> users.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>> > > >>>>>> pre-discussion
>> > > >>>>>>>> [1]
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>> Please
>> > > >>>>>>>>>>>>>>>>>>>>>>> correct
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> me
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
>> > > >>>>> conclusion
>> > > >>>>>>> of
>> > > >>>>>>>> that
>> > > >>>>>>>>>>>>>>> discussion
>> > > >>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>> use
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
>> > > >>>>>> interface
>> > > >>>>>>>> and
>> > > >>>>>>>>>>>>> implement
>> > > >>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>> different
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> stores
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> such
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
>> > > >>>> external
>> > > >>>>>>>> system.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
>> > > >>>>> between
>> > > >>>>>>> the
>> > > >>>>>>>>>>> current
>> > > >>>>>>>>>>>>> FLIP
>> > > >>>>>>>>>>>>>>>>> design
>> > > >>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
>> > > >>> of
>> > > >>>>> the
>> > > >>>>>>>>>> discussion
>> > > >>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>> thread
>> > > >>>>>>>>>>>>>>>>>> [1] ?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>
>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>> > > >>>>> 3:41 PM
>> > > >>>>>>>> yuxia <
>> > > >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
>> > > >>> driving
>> > > >>>>> this
>> > > >>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>> > > >>>>> handling
>> > > >>>>>>>> catalog
>> > > >>>>>>>>>>> with
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> same
>> > > >>>>>>>>>>>>>>>>>> name
>> > > >>>>>>>>>>>>>>>>>>>>>> looks a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> little
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
>> > > >>>>> it'll
>> > > >>>>>> be
>> > > >>>>>>>>>> better
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> explain
>> > > >>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> java
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> doc
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
>> > > >>> give a
>> > > >>>>>> brief
>> > > >>>>>>>>>> example
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>> FLIP.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
>> > > >>> name
>> > > >>>>>>>> `addCatalog`
>> > > >>>>>>>>>>>> still
>> > > >>>>>>>>>>>>>>> looks
>> > > >>>>>>>>>>>>>>>>>>> confused
>> > > >>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> me.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
>> > > >>>> storing
>> > > >>>>>>>> catalog to
>> > > >>>>>>>>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>>>>>>>> how
>> > > >>>>>>>>>>>>>>>>>>>>> about
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
>> > > >>> It's
>> > > >>>>> very
>> > > >>>>>>>> personal
>> > > >>>>>>>>>>>>>> opinion,
>> > > >>>>>>>>>>>>>>>> you
>> > > >>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>> decide
>> > > >>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> take
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
>> > > >>>>>>> statement,
>> > > >>>>>>>>>> which
>> > > >>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>> called?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>> > > >>>>>>>> `addCatalog`?
>> > > >>>>>>>>>> I'm
>> > > >>>>>>>>>>>>>>> wondering
>> > > >>>>>>>>>>>>>>>>>>> whether
>> > > >>>>>>>>>>>>>>>>>>>>> user
>> > > >>>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> add a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
>> > > >>> SQL
>> > > >>>>>>>> stement.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
>> > > >>>> neccessary
>> > > >>>>> to
>> > > >>>>>>>>>> provide a
>> > > >>>>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>>>>>> implmentation
>> > > >>>>>>>>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>> > > >>>>>>> `CatalogStoreFactory`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>> > > >>>>>>>> registration
>> > > >>>>>>>>>> for
>> > > >>>>>>>>>>>>>> catalog.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
>> > > >>> catalog
>> > > >>>>>> with
>> > > >>>>>>>> CREATE
>> > > >>>>>>>>>>>>>> CATALOG,
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>> > > >>>> used
>> > > >>>>>> by
>> > > >>>>>>>>>> default.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
>> > > >>>>> registration
>> > > >>>>>> is
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>> behavior,
>> > > >>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>> there
>> > > >>>>>>>>>>>>>>>>>>>>>>>> any
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> way
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
>> > > >>> to
>> > > >>>>>>>> synchronous
>> > > >>>>>>>>>>>>>>> registration
>> > > >>>>>>>>>>>>>>>>> just
>> > > >>>>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>>>>>>> before?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
>> > > >>>>> `addCatalog`
>> > > >>>>>>> and
>> > > >>>>>>>>>>>>>>>> `registerCatalog`
>> > > >>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
>> > > >>>>>>>> registration,
>> > > >>>>>>>>>> it
>> > > >>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>> well
>> > > >>>>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
>> > > >>>> but
>> > > >>>>>> then
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>> following
>> > > >>>>>>>>>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>>>>>>>> TABLE
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> statement
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
>> > > >>> fail
>> > > >>>>> to
>> > > >>>>>>>> open.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
>> > > >>>> change
>> > > >>>>>>> which
>> > > >>>>>>>>>>> should
>> > > >>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>> highlighted
>> > > >>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> may
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
>> > > >>> part.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
>> > > >>>> asynchronous
>> > > >>>>>>>>>>>> registration, I
>> > > >>>>>>>>>>>>>>> would
>> > > >>>>>>>>>>>>>>>>>> like
>> > > >>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>> expect
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> there
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
>> > > >>>> open
>> > > >>>>>> or
>> > > >>>>>>>>>> register
>> > > >>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> background,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
>> > > >>> comments,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
>> > > >>>> will
>> > > >>>>> be
>> > > >>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>> has
>> > > >>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
>> > > >>>> initialization
>> > > >>>>>>>> process
>> > > >>>>>>>>>>> fails,
>> > > >>>>>>>>>>>>>> these
>> > > >>>>>>>>>>>>>>>>>>>> statements
>> > > >>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
>> > > >>> successfully."
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
>> > > >>> lazy
>> > > >>>>>>>>>> initialization
>> > > >>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>> than
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
>> > > >>>>>>>>>> jinfeng1995@gmail.com
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
>> > > >>>>>>> dev@flink.apache.org
>> > > >>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
>> > > >>> 03日
>> > > >>>> 下午
>> > > >>>>>>>> 3:27:45
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
>> > > >>>> 295:
>> > > >>>>>>>> Support
>> > > >>>>>>>>>>>>>> persistence
>> > > >>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
>> > > >>>>> registration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
>> > > >>>> all
>> > > >>>>>> for
>> > > >>>>>>>> your
>> > > >>>>>>>>>>>>> interest
>> > > >>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>> DISCUSS.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
>> > > >>> catalog
>> > > >>>>>> with
>> > > >>>>>>>> the
>> > > >>>>>>>>>> same
>> > > >>>>>>>>>>>>> name
>> > > >>>>>>>>>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>>>>> for
>> > > >>>>>>>>>>>>>>>>>>>>>>> both
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
>> > > >>>> crucial
>> > > >>>>>>>> point.
>> > > >>>>>>>>>>> Based
>> > > >>>>>>>>>>>> on
>> > > >>>>>>>>>>>>>> my
>> > > >>>>>>>>>>>>>>>>>> current
>> > > >>>>>>>>>>>>>>>>>>>>>> personal
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
>> > > >>>>>> Map<String,
>> > > >>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>> serve
>> > > >>>>>>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>>>>> a
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cache
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
>> > > >>> catalogs
>> > > >>>>> and
>> > > >>>>>>>> have
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>> highest
>> > > >>>>>>>>>>>>>>>>>>> priority.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
>> > > >>> methods
>> > > >>>>> that
>> > > >>>>>>> can
>> > > >>>>>>>>>> affect
>> > > >>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
>> > > >>> registerCatalog(String
>> > > >>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
>> > > >>>>> catalog
>> > > >>>>>>>> instance
>> > > >>>>>>>>>>>> into
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>> > > >>>> unregisterCatalog(String
>> > > >>>>>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>> > > >>> the
>> > > >>>>>> catalog
>> > > >>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>> corresponding
>> > > >>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
>> > > >>>>> Catalog>
>> > > >>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
>> > > >>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
>> > > >>>> retrieves
>> > > >>>>>> the
>> > > >>>>>>>>>>>>> corresponding
>> > > >>>>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
>> > > >>> Catalog>
>> > > >>>>>>>> catalogs. If
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>>>> does
>> > > >>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>> exist,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
>> > > >>>> corresponding
>> > > >>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>> from
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
>> > > >>> puts
>> > > >>>>> the
>> > > >>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>> into
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>> > > >>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
>> > > >>> methods
>> > > >>>>>> only
>> > > >>>>>>>> modify
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
>> > > >>>>>>> catalogName,
>> > > >>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>> String>
>> > > >>>>>>>>>>>>>>>>>>>>>>> properties)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
>> > > >>>>>>> properties
>> > > >>>>>>>> to
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> catalogStore
>> > > >>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>> checks
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
>> > > >>>>>> catalogName
>> > > >>>>>>>> with
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>>> same
>> > > >>>>>>>>>>>>>>>> name.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>> > > >>> removeCatalog(String
>> > > >>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>> > > >>> the
>> > > >>>>>>> specified
>> > > >>>>>>>>>>>>>> configuration
>> > > >>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> specified
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
>> > > >>>>>>> catalogStore.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
>> > > >>>> possible
>> > > >>>>>>>> conflict
>> > > >>>>>>>>>>>>>> scenarios:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
>> > > >>> corresponding
>> > > >>>>>>>> catalogName
>> > > >>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
>> > > >>> in
>> > > >>>> the
>> > > >>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>> Catalog>,
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>> > > >>>>>>>> catalogName,
>> > > >>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>> can
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
>> > > >>> directly
>> > > >>>>> saved
>> > > >>>>>>> to
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
>> > > >>> corresponding
>> > > >>>>>>>> catalogName
>> > > >>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>> both
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
>> > > >>>>>>> Map<String,
>> > > >>>>>>>>>>>> Catalog>,
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
>> > > >>>>> catalog)
>> > > >>>>>>>> method
>> > > >>>>>>>>>>> will
>> > > >>>>>>>>>>>>>> fail.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
>> > > >>> corresponding
>> > > >>>>>>>> catalogName
>> > > >>>>>>>>>>>>> already
>> > > >>>>>>>>>>>>>>>>> exists
>> > > >>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
>> > > >>> the
>> > > >>>>>>>>>>>> addCatalog(String
>> > > >>>>>>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
>> > > >>>> method
>> > > >>>>>> can
>> > > >>>>>>>>>> directly
>> > > >>>>>>>>>>>>> save
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> properties
>> > > >>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
>> > > >>>>>>>>>> getCatalog(String
>> > > >>>>>>>>>>>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
>> > > >>>> for
>> > > >>>>>>>>>>> initialization
>> > > >>>>>>>>>>>>>>> because
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> corresponding
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
>> > > >>> already
>> > > >>>>>> exists
>> > > >>>>>>>> in
>> > > >>>>>>>>>>>> catalogs
>> > > >>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
>> > > >>>>>>>>>>>>> unregisterCatalog(String
>> > > >>>>>>>>>>>>>>>>>>>> catalogName)
>> > > >>>>>>>>>>>>>>>>>>>>>>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
>> > > >>>>>>>> corresponding to
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>> original
>> > > >>>>>>>>>>>>>>>>>>>>> catalogName
>> > > >>>>>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
>> > > >>> confuse
>> > > >>>>>> users
>> > > >>>>>>>> that
>> > > >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
>> > > >>>>> catalog)`
>> > > >>>>>>> in
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>>>> `Map<String,
>> > > >>>>>>>>>>>>>>>>>>>> Catalog>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>> `registerCatalog(String
>> > > >>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>> String>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
>> > > >>>> lead
>> > > >>>>>> to
>> > > >>>>>>>>>>>> confusion. I
>> > > >>>>>>>>>>>>>>>> suggest
>> > > >>>>>>>>>>>>>>>>>>>>> changing
>> > > >>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
>> > > >>> to
>> > > >>>>>>>>>>> addCatalog(String
>> > > >>>>>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
>> > > >>> as
>> > > >>>>>>>> previously
>> > > >>>>>>>>>>>>> mentioned
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
>> > > >>>>> `registerCatalog(String
>> > > >>>>>>>>>>>>>>> catalogName,Catalog
>> > > >>>>>>>>>>>>>>>>>>>> catalog,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
>> > > >>> method
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
>> > > >>>> already
>> > > >>>>> an
>> > > >>>>>>>>>> instance,
>> > > >>>>>>>>>>>>>> adding
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>> "lazyInit"
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
>> > > >>>>>>>>>> registerCatalog(String
>> > > >>>>>>>>>>>>>>>>> catalogName,
>> > > >>>>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
>> > > >>>> necessarily
>> > > >>>>>>>> result in
>> > > >>>>>>>>>>> lazy
>> > > >>>>>>>>>>>>>>>>>>> initialization.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
>> > > >>>> about
>> > > >>>>>>>>>> encryption
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
>> > > >>>>>>> necessary,
>> > > >>>>>>>> but
>> > > >>>>>>>>>>>>> perhaps
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> encryption
>> > > >>>>>>>>>>>>>>>>>>>>>>> logic
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
>> > > >>> in
>> > > >>>>> the
>> > > >>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>> when
>> > > >>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> data
>> > > >>>>>>>>>>>>>>>>>>>> is
>> > > >>>>>>>>>>>>>>>>>>>>>>>> actually
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
>> > > >>> the
>> > > >>>>>>>>>>> FileCatalogStore
>> > > >>>>>>>>>>>>>> could
>> > > >>>>>>>>>>>>>>>>>> encrypt
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>> data
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> when
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
>> > > >>>>> while
>> > > >>>>>>> the
>> > > >>>>>>>>>>>>>>>> MemoryCatalogStore
>> > > >>>>>>>>>>>>>>>>>>> does
>> > > >>>>>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> require
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
>> > > >>> the
>> > > >>>>>>>>>>>>>> `MemoryCatalogStore`?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
>> > > >>> necessary
>> > > >>>> to
>> > > >>>>>>> have a
>> > > >>>>>>>>>>>>>>>>> MemoryCatalogStore
>> > > >>>>>>>>>>>>>>>>>> as
>> > > >>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> default
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
>> > > >>>> saves
>> > > >>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>> configurations
>> > > >>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>> memory.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
>> > > >>> to
>> > > >>>>>>>> implement
>> > > >>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>> loading
>> > > >>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> we
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
>> > > >>> introduce
>> > > >>>>>>>> additional
>> > > >>>>>>>>>>> cache.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
>> > > >>>>>>>> registration
>> > > >>>>>>>>>>>> failed?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
>> > > >>> critical
>> > > >>>>>>> concern.
>> > > >>>>>>>>>> When
>> > > >>>>>>>>>>>>>>> executing
>> > > >>>>>>>>>>>>>>>>> DDL,
>> > > >>>>>>>>>>>>>>>>>>>> DQL,
>> > > >>>>>>>>>>>>>>>>>>>>> or
>> > > >>>>>>>>>>>>>>>>>>>>>>> DML
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
>> > > >>>> reference a
>> > > >>>>>>>> specific
>> > > >>>>>>>>>>>>> Catalog,
>> > > >>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>> Catalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
>> > > >>> if it
>> > > >>>>> has
>> > > >>>>>>> not
>> > > >>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>> initialized
>> > > >>>>>>>>>>>>>>>>>> yet.
>> > > >>>>>>>>>>>>>>>>>>>> If
>> > > >>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
>> > > >>>>> fails,
>> > > >>>>>>>> these
>> > > >>>>>>>>>>>>>> statements
>> > > >>>>>>>>>>>>>>>> will
>> > > >>>>>>>>>>>>>>>>>> not
>> > > >>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>> executed
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
>> > > >>>>>> Catalog>
>> > > >>>>>>>>>> catalogs
>> > > >>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
>> > > >>>> have
>> > > >>>>>> the
>> > > >>>>>>>> same
>> > > >>>>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>>>> name,
>> > > >>>>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>>>> correponding
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>> > > >>>>> instances.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
>> > > >>>> resolved
>> > > >>>>> by
>> > > >>>>>>>>>> referring
>> > > >>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> previous
>> > > >>>>>>>>>>>>>>>>>>>>>>>> responses.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
>> > > >>> the
>> > > >>>>>>> Catalog
>> > > >>>>>>>>>> that
>> > > >>>>>>>>>>>> has
>> > > >>>>>>>>>>>>>> been
>> > > >>>>>>>>>>>>>>>>> most
>> > > >>>>>>>>>>>>>>>>>>>>> recently
>> > > >>>>>>>>>>>>>>>>>>>>>>>> used
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
>> > > >>>> for
>> > > >>>>>>>> subsequent
>> > > >>>>>>>>>>>> use.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
>> > > >>>>>> 10:58 PM
>> > > >>>>>>>>>> Xianxun
>> > > >>>>>>>>>>>> Ye
>> > > >>>>>>>>>>>>> <
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
>> > > >>> this
>> > > >>>>>> Flip,
>> > > >>>>>>>> I do
>> > > >>>>>>>>>>>>> believe
>> > > >>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>> Flip
>> > > >>>>>>>>>>>>>>>>>>>>> could
>> > > >>>>>>>>>>>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
>> > > >>>> work
>> > > >>>>>> for
>> > > >>>>>>>> also
>> > > >>>>>>>>>>>>> manages a
>> > > >>>>>>>>>>>>>>> lot
>> > > >>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>> catalogs,
>> > > >>>>>>>>>>>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
>> > > >>>> becomes
>> > > >>>>>>> slow
>> > > >>>>>>>>>>> because
>> > > >>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>> loading a
>> > > >>>>>>>>>>>>>>>>>>>>> number
>> > > >>>>>>>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
>> > > >>> catalogs in
>> > > >>>>> the
>> > > >>>>>>>> user's
>> > > >>>>>>>>>>>> Flink
>> > > >>>>>>>>>>>>>> SQL
>> > > >>>>>>>>>>>>>>>>>> through
>> > > >>>>>>>>>>>>>>>>>>>>>> regular
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
>> > > >>>>> catalogs
>> > > >>>>>>> to
>> > > >>>>>>>>>>> improve
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>> speed
>> > > >>>>>>>>>>>>>>>>> of
>> > > >>>>>>>>>>>>>>>>>>>> task
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
>> > > >>> flip-295, I
>> > > >>>>>> have
>> > > >>>>>>>> some
>> > > >>>>>>>>>>>>>> questions:
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
>> > > >>>>> catalog
>> > > >>>>>>> with
>> > > >>>>>>>>>>> CREATE
>> > > >>>>>>>>>>>>>>> CATALOG,
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>> > > >>>> used
>> > > >>>>>> by
>> > > >>>>>>>>>> default.
>> > > >>>>>>>>>>>>> What
>> > > >>>>>>>>>>>>>> if
>> > > >>>>>>>>>>>>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
>> > > >>>>> implement
>> > > >>>>>>>>>>> asynchronous
>> > > >>>>>>>>>>>>>>>>>> registration?
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
>> > > >>> same
>> > > >>>>>>>> question
>> > > >>>>>>>>>> with
>> > > >>>>>>>>>>>>>>> Shammon’s
>> > > >>>>>>>>>>>>>>>>>> first
>> > > >>>>>>>>>>>>>>>>>>>>>>> comment.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> The
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>> > > >>>>> catalogs
>> > > >>>>>>> and
>> > > >>>>>>>>>>>>>> CatalogStore
>> > > >>>>>>>>>>>>>>>>>>>> catalogstore
>> > > >>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
>> > > >>> have
>> > > >>>> the
>> > > >>>>>>> same
>> > > >>>>>>>>>>> catalog
>> > > >>>>>>>>>>>>>> name,
>> > > >>>>>>>>>>>>>>>> but
>> > > >>>>>>>>>>>>>>>>>>>>>> correponding
>> > > >>>>>>>>>>>>>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>> > > >>>>> instances.
>> > > >>>>>>> For
>> > > >>>>>>>>>>> example:
>> > > >>>>>>>>>>>>>>>>> catalog1->
>> > > >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
>> > > >>>>>>>>>>>>>>>>>>>>>>>> in
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
>> > > >>> catalog1 ->
>> > > >>>>>>>> hiveCatalog
>> > > >>>>>>>>>>> in
>> > > >>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> catalogstore.
>> > > >>>>>>>>>>>>>>>>>>>>>>> Would
>> > > >>>>>>>>>>>>>>>>>>>>>>>>> this
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
>> > > >>> should
>> > > >>>>>> define
>> > > >>>>>>>>>>> `catalogs`
>> > > >>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>>>> `catalogstore`
>> > > >>>>>>>>>>>>>>>>>>>>>>> more
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
>> > > >>> 11:13，Hang
>> > > >>>>>> Ruan
>> > > >>>>>>> <
>> > > >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> 写道：
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > >
>> > >
>> >
>>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
hi, Hang
Thanks for your reply, Make sense to me, I updated the FLIP.
Best,
Feng
On Fri, Jun 9, 2023 at 3:34 PM Hang Ruan <ru...@gmail.com> wrote:
> Hi, Feng.
>
> I reviewed this FLIP again and found some little places that may need to
> optimize.
>
> 1. `CatalogDescriptor` may need a private constructor.
> 2. The method `initCatalog` in `CatalogManager` is not introduced.
>
> Best,
> Hang
>
> Feng Jin <ji...@gmail.com> 于2023年6月6日周二 21:17写道：
>
> > Hi Leonard,
> >
> > Thanks for your reply.
> >
> > > 1. a How to construct a CatalogDescriptor ?
> >
> > I think it would be helpful to add a method for constructing a
> > CatalogDescriptor, as you mentioned in 1.c. I will update the
> documentation
> > later.
> >
> > > 1.b How to visit the fields ? Could we use Configuration instead of
> > Map<String, String> ?
> >
> > I believe that the use of Map<String, String> options is only intended
> for
> > creating a catalog and not for accessing internal parameters.
> >
> > Since all of the relevant parameters for CREATE CATALOG are also stored
> in
> > Map<String, String> options, my understanding is that using Map<String,
> > String> options should suffice.
> >
> > Here is the implementation of execute CREATE CATALOG statement.
> > ```java
> > private TableResultInternal createCatalog(CreateCatalogOperation
> operation)
> > {
> > String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
> > try {
> > String catalogName = operation.getCatalogName();
> > Map<String, String> properties = operation.getProperties();
> >
> > Catalog catalog =
> > FactoryUtil.createCatalog(
> > catalogName,
> > properties,
> > tableConfig,
> > resourceManager.getUserClassLoader());
> > catalogManager.registerCatalog(catalogName, catalog);
> >
> > return TableResultImpl.TABLE_RESULT_OK;
> > } catch (CatalogException e) {
> > throw new ValidationException(exMsg, e);
> > }
> > }
> > ```
> >
> >
> > > 2. Do we have plan to offer a default CatalogStore if user didn’t
> config
> > this?
> >
> > Yes, the in-memory catalogStore will be used as the default CatalogStore
> > even if the user has not configured one
> >
> >
> > Best,
> > Feng
> >
> >
> > On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
> >
> > > Hi, Feng
> > >
> > > Sorry for reply late, but I’ve some comments about the FLIP
> > >
> > >
> > > 1. The introduced Public class CatalogDescriptor seems missed some
> > > necessary component
> > > a) How to construct a CatalogDescriptor ?
> > > b) How to visit the fields ? Could we use Configuration instead of
> > > Map<String, String> ?
> > > c) Could we offer a built-in factory method to build a
> > CatalogDescriptor
> > > like
> > > public static CatalogDescriptor of(String catalogName,
> Configuration
> > > configuration)
> > >
> > > 2. The FLIP said “By default, there are two built-in CatalogStores
> > > available: the In-Memory CatalogStore and the File CatalogStore” ,
> > > Do we have plan to offer a default CatalogStore if user didn’t config
> > > this? IIUC, users can obtains the benefits from lazy catalog
> > > initialization If
> > > we have a default catalogstore even it is in-memory.
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > >
> > > > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > Thank you everyone for your valuable input. If there are no further
> > > > questions or concerns about the FLIP[1], I would like to start voting
> > > > tomorrow (6/7).
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > >
> > > > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > >> Hi Samrat,
> > > >>
> > > >> Thanks for your advice.
> > > >>
> > > >>> 1. The createCatalog method does not mention any exceptions being
> > > >> thrown.
> > > >>
> > > >> CreateCatalog will throw CatalogException like registerCatalog. As
> > > >> CatalogException is a RuntimeException,
> > > >> there is no explicit declaration of throwing Exceptions in
> > > CatalogManager
> > > >> and TableEnvironment.
> > > >> To avoid misunderstandings, I have added the "throw
> CatalogException"
> > > flag
> > > >> to the createCatalog method definition of CatalogStore.
> > > >>
> > > >>> 2. Could you please provide an exhaustive list of the supported
> > kinds?
> > > >>
> > > >> Sure, the documentation now includes the configuration of the
> > built-in
> > > >> CatalogStore as well as how to configure a custom CatalogStore.
> > > >>
> > > >>
> > > >> Best,
> > > >> Feng
> > > >>
> > > >>
> > > >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi Feng,
> > > >>>
> > > >>> Thank you for providing the proposal. I believe this feature will
> be
> > > >>> highly
> > > >>> valuable.
> > > >>>
> > > >>> I have a couple of inquiries:
> > > >>>
> > > >>> 1. According to the documentation [1], the createCatalog method
> does
> > > not
> > > >>> mention any exceptions being thrown. However, I would like to
> confirm
> > > if
> > > >>> it
> > > >>> is always true that there will be no failures in all scenarios.
> > Please
> > > let
> > > >>> me know if there is any additional information I may have missed.
> > > >>>
> > > >>> 2. Regarding the registration process using the
> > > `table.catalog-store.kind`
> > > >>> configuration, could you please provide an exhaustive list of the
> > > >>> supported
> > > >>> kinds?
> > > >>> It would be great to have a comprehensive understanding of the
> > > options
> > > >>> available.
> > > >>>
> > > >>> Bests,
> > > >>> Samrat
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >>>
> > > >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com>
> > wrote:
> > > >>>
> > > >>>> Hi Hang and Jark
> > > >>>>
> > > >>>> Thank you very much for your reply.
> > > >>>>
> > > >>>> @Jark
> > > >>>>> 1. Could you move the CatalogStore registration API to the
> "Public
> > > >>>> Interface" section?
> > > >>>>
> > > >>>> Indeed, it is more reasonable.
> > > >>>>
> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > > >>>>
> > > >>>> Sure, the names that you provided are indeed more logical and
> > easier
> > > >>> to
> > > >>>> understand.
> > > >>>>
> > > >>>>> 3. About the open/close method in CatalogStoreFactory.
> > > >>>>
> > > >>>> I also agree with the proposed requirement scenario and design. I
> > have
> > > >>>> already made modifications to the interface.
> > > >>>>
> > > >>>> @Hang
> > > >>>>
> > > >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > > close
> > > >>>> the resource.
> > > >>>>
> > > >>>> I have already added the missing method.
> > > >>>>
> > > >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > > >>>>
> > > >>>> Sorry for the misspelling, I updated this part.
> > > >>>>
> > > >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> > > methods
> > > >>>> to init or close its resource
> > > >>>>
> > > >>>> This is indeed a scenario that needs to be considered, and I have
> > > added
> > > >>>> relevant methods accordingly.
> > > >>>> Additionally, I have included some relevant description
> information
> > in
> > > >>> the
> > > >>>> documentation to help others understand
> > > >>>> why the open/close methods need to be added.
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>> Feng
> > > >>>>
> > > >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> > > >>>>
> > > >>>>> Hi Jing,
> > > >>>>>
> > > >>>>> Thank you for the update.
> > > >>>>>
> > > >>>>> 1. Could you move the CatalogStore registration API to the
> "Public
> > > >>>>> Interface" section?
> > > >>>>> "Proposed Changes" is more like a place to describe the
> > > implementation
> > > >>>>> details.
> > > >>>>>
> > > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > > >>> Besides,
> > > >>>>> the config key
> > > >>>>> name should be hierarchical[1]. Therefore, it may be better to
> use:
> > > >>>>> "table.catalog-store.kind"
> > > >>>>> "table.catalog-store.file.path"
> > > >>>>>
> > > >>>>> 3. I think Hang's suggestions make sense.
> > > >>>>>
> > > >>>>> Others look good to me.
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Jark
> > > >>>>>
> > > >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
> > > >>> wrote:
> > > >>>>>
> > > >>>>>> Hi, Feng.
> > > >>>>>>
> > > >>>>>> Thanks for the update.
> > > >>>>>> The current design looks good to me. I have some minor comments.
> > > >>>>>>
> > > >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > > >>> close
> > > >>>>> the
> > > >>>>>> resource. For example, when storing the information in MySQL,
> the
> > > >>> store
> > > >>>>>> needs to open and close the connections.
> > > >>>>>>
> > > >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > > >>>>>>
> > > >>>>>> 3. About the usage in the sql gateway.
> > > >>>>>> Considering the usage in sql gateway, the sql gateway may
> create a
> > > >>>>>> CatalogStore for each session.
> > > >>>>>> If we are using the MySqlCatalogStore, there would be so many
> > > >>>>> connections.
> > > >>>>>> How can we reuse the connection among these sessions?
> > > >>>>>> I think sql gateway need to maintain a connection pool in
> > > >>>>>> the CatalogStoreFactory and each session get its own connection
> > from
> > > >>>> the
> > > >>>>>> pool when it is created.
> > > >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close`
> methods
> > > >>> to
> > > >>>>> init
> > > >>>>>> or close its resource.
> > > >>>>>> Or is there a better way?
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Hang
> > > >>>>>>
> > > >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > > >>>>>>
> > > >>>>>>> Thanks Jingsong.
> > > >>>>>>>
> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> > > >>>>>>>
> > > >>>>>>> +1 For this, I have already updated FLIP.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Feng
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> > > >>> jingsonglee0@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Thanks Feng,
> > > >>>>>>>>
> > > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > >>>>>>>> functions are converted to createxxx from registerxxx.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <
> jinfeng1995@gmail.com
> > > >>>>
> > > >>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Hi jark, thanks for your suggestion.
> > > >>>>>>>>>
> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > > >>> the
> > > >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> > > >>>>> Otherwise,
> > > >>>>>>>> there
> > > >>>>>>>>> might be some data inconsistencies when CatalogStore is
> > > >>> changed.
> > > >>>>>>>>>
> > > >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> > > >>>>>> TableEnv.
> > > >>>>>>>>> Therefore, my current proposal is to add a method to
> configure
> > > >>>> the
> > > >>>>>>>>> CatalogStore in EnvironmentSettings.
> > > >>>>>>>>>
> > > >>>>>>>>> // Initialize a catalog Store
> > > >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> > > >>>>>>>>>
> > > >>>>>>>>> // set up the Table API
> > > >>>>>>>>> final EnvironmentSettings settings =
> > > >>>>>>>>>
> > > >>> EnvironmentSettings.newInstance().inBatchMode()
> > > >>>>>>>>> .withCatalogStore(catalogStore)
> > > >>>>>>>>> .build();
> > > >>>>>>>>>
> > > >>>>>>>>> final TableEnvironment tableEnv =
> > > >>>>>>>> TableEnvironment.create(settings);
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > > >>>> default
> > > >>>>>>>> method,
> > > >>>>>>>>> not an interface method?
> > > >>>>>>>>>
> > > >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> > > >>> that
> > > >>>> as
> > > >>>>> a
> > > >>>>>>> new
> > > >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> > > >>>> method
> > > >>>>>> but
> > > >>>>>>>> an
> > > >>>>>>>>> interface method. I have already modified the interface.
> > > >>>>>>>>>
> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > > >>>>>>>>> deprecated`registerCatalog`.
> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > > >>> mention
> > > >>>> the
> > > >>>>>>>> changed
> > > >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > > >>>> other
> > > >>>>>>>>> persisted catalog store) is used.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> [1].
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Feng
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>
> > > >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> > > >>>>>>>>>> The current design looks pretty good to me. I just have some
> > > >>>>> minor
> > > >>>>>>>>>> comments.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > > >>> the
> > > >>>>>>>> CatalogStore
> > > >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> > > >>> there
> > > >>>>>> might
> > > >>>>>>> be
> > > >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > > >>>> default
> > > >>>>>>>> method,
> > > >>>>>>>>>> not an interface method?
> > > >>>>>>>>>>
> > > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > > >>>>> deprecated
> > > >>>>>>>>>> `registerCatalog`.
> > > >>>>>>>>>>
> > > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > > >>> mention
> > > >>>> the
> > > >>>>>>>> changed
> > > >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > > >>>>> other
> > > >>>>>>>>>> persisted catalog store) is used.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Jark
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> > > >>> jinfeng1995@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> @Ron
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> > > >>>> method,
> > > >>>>>> do
> > > >>>>>>> we
> > > >>>>>>>>>> need
> > > >>>>>>>>>>> to provide a default implementation?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yes, we will provide a default
> > > >>> InMemoryCatalogStoreFactory to
> > > >>>>>>> create
> > > >>>>>>>> an
> > > >>>>>>>>>>> InMemoryCatalogStore.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> > > >>> initializing
> > > >>>>> it,
> > > >>>>>>>> whether
> > > >>>>>>>>>> we
> > > >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> > > >>> snapshots,
> > > >>>>> and
> > > >>>>>>>> once
> > > >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> > > >>>>>> Catalog>
> > > >>>>>>>>>>> catalogs.
> > > >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> > > >>> the
> > > >>>>>> primary
> > > >>>>>>>>>> source
> > > >>>>>>>>>>> for obtaining the corresponding Catalog.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> > > >>>>>>>>>>> `catalog.store.path`?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> > > >>> seem a
> > > >>>>> bit
> > > >>>>>>>>>> strange. I
> > > >>>>>>>>>>> will update the FLIP.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thank you for the review. I have made the necessary
> > > >>>>> corrections.
> > > >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> > > >>> I
> > > >>>>> have
> > > >>>>>>> also
> > > >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> Feng
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> > > >>>> zjureel@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi feng,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for updating, I have some minor comments
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> > > >>>>>> `Public
> > > >>>>>>>>>>>> Interfaces`, it is not a public interface.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> > > >>> and
> > > >>>>>>>>>>>> `CatalogStoreFactory`
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> > > >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> > > >>>>> should
> > > >>>>>> be
> > > >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> > > >>>>>>>>>>>> catalogStore.get(catalogName);`
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best,
> > > >>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> > > >>>> ron9.liu@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Feng
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> > > >>>> useful
> > > >>>>>> for
> > > >>>>>>>>>> catalog
> > > >>>>>>>>>>>>> management.
> > > >>>>>>>>>>>>> I have some small questions:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> 1. Regarding the
> > > >>> CatalogStoreFactory#createCatalogStore
> > > >>>>>> method,
> > > >>>>>>>> do we
> > > >>>>>>>>>>>> need
> > > >>>>>>>>>>>>> to provide a default implementation?
> > > >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> > > >>>> initializing
> > > >>>>>> it,
> > > >>>>>>>>>> whether
> > > >>>>>>>>>>> we
> > > >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> > > >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> > > >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> > > >>> them to
> > > >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>> Ron
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> > > >>> 21:19写道：
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi yuxia
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> > > >>>>> register
> > > >>>>>>> the
> > > >>>>>>>>>>> Catalog,
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>> initialize it and open it. right?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> > > >>>>>> original
> > > >>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> > > >>>>>>>>>> registerCatalog
> > > >>>>>>>>>>> so
> > > >>>>>>>>>>>>>> that parameter validation can be performed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> > > >>>>>>> reflected
> > > >>>>>>>> in
> > > >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> > > >>>>> catalog
> > > >>>>>>>>>>>>> configurations,
> > > >>>>>>>>>>>>>> only initialization is required in getCatalog.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi, Feng.
> > > >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> > > >>>>>>>> initialization*.
> > > >>>>>>>>>>> If
> > > >>>>>>>>>>>>> i'm
> > > >>>>>>>>>>>>>>> wrong, please correct me.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> > > >>>> access
> > > >>>>>> the
> > > >>>>>>>>>>> catalog,
> > > >>>>>>>>>>>>> then
> > > >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> > > >>>>> Changes,
> > > >>>>>>>> once we
> > > >>>>>>>>>>>>>> register
> > > >>>>>>>>>>>>>>> the Catalog,
> > > >>>>>>>>>>>>>>> we initialize it and open it. right?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>> Yuxia
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> ----- 原始邮件 -----
> > > >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
> > > >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > >>>>>> Catalog
> > > >>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>> and asynchronous registration
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> One of the major changes is that current design
> > > >>> will
> > > >>>>>>> provide
> > > >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> > > >>>> of a
> > > >>>>>>>> cache,
> > > >>>>>>>>>>> which
> > > >>>>>>>>>>>>>> means
> > > >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> > > >>>> other
> > > >>>>>>>> sessions
> > > >>>>>>>>>>> will
> > > >>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> > > >>>>>> further
> > > >>>>>>>>>>>> improvement.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > >>>>>>>> jinfeng1995@gmail.com>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> > > >>>>>> progress
> > > >>>>>>>> of
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> > > >>> Shammon
> > > >>>> FY,
> > > >>>>>>>> ShengKai
> > > >>>>>>>>>>> Fang
> > > >>>>>>>>>>>>>> and I
> > > >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> > > >>>>>> solution
> > > >>>>>>>> for
> > > >>>>>>>>>>> Flink
> > > >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> > > >>>> have
> > > >>>>>>>> updated
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> final
> > > >>>>>>>>>>>>>>>> solution in FLIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 1.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> > > >>>>>>>> configuration
> > > >>>>>>>>>>>> similar
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>> TableDescriptor.
> > > >>>>>>>>>>>>>>>> 2.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> > > >>>>>>>>>>> storeCatalog(String
> > > >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> > > >>>>>> CatalogDescriptor
> > > >>>>>>>>>>>>>>> getCatalog(String)
> > > >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> > > >>>> Catalog
> > > >>>>>>>> instance.
> > > >>>>>>>>>>>> This
> > > >>>>>>>>>>>>>> way,
> > > >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> > > >>> saving
> > > >>>>> and
> > > >>>>>>>>>>> retrieving
> > > >>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> configurations without having to initialize
> > > >>>>>> catalogs.
> > > >>>>>>>>>>>>>>>> 3.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The default registerCatalog(String
> > > >>> catalogName,
> > > >>>>>>> Catalog
> > > >>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> > > >>>>>>> deprecated.
> > > >>>>>>>>>>>>>>>> 4.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> A new function registerCatalog(String
> > > >>>> catalogName,
> > > >>>>>>>>>>>>> CatalogDescriptor
> > > >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> > > >>> default
> > > >>>>>>>> registration
> > > >>>>>>>>>>>>> function
> > > >>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> > > >>>>>>>>>>>>>>>> 5.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> > > >>>> will
> > > >>>>>>>> remain
> > > >>>>>>>>>>>>> unchanged
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> > > >>>> deletion
> > > >>>>>>>>>> operations
> > > >>>>>>>>>>>> from
> > > >>>>>>>>>>>>>> one
> > > >>>>>>>>>>>>>>>> session won't synchronize with other
> > > >>> sessions.
> > > >>>>>>>>>>>>>>>> 6.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> To support multi-session synchronization
> > > >>>> scenarios
> > > >>>>>> for
> > > >>>>>>>>>>> deletions
> > > >>>>>>>>>>>>>> later
> > > >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> > > >>>>>>>>>>> configurable.There
> > > >>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>> three possible situations:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> > > >>>>>>>> automatically
> > > >>>>>>>>>>> clear
> > > >>>>>>>>>>>>>>>> long-unused catalogs.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> > > >>>>>> getCatalog
> > > >>>>>>>>>>> creates a
> > > >>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>> instance.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> This is the document for discussion:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> This is the final proposal document:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thank you very much for your attention and
> > > >>>>> suggestions
> > > >>>>>> on
> > > >>>>>>>> this
> > > >>>>>>>>>>>>> FLIP. A
> > > >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> > > >>> on
> > > >>>>> the
> > > >>>>>>>> entire
> > > >>>>>>>>>>>> design
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>> organizing offline discussions.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> > > >>>> about
> > > >>>>>>> this
> > > >>>>>>>> FLIP
> > > >>>>>>>>>>>>> please
> > > >>>>>>>>>>>>>>> feel
> > > >>>>>>>>>>>>>>>> free to share.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > >>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> > > >>> to
> > > >>>> me.
> > > >>>>>> We
> > > >>>>>>>> could
> > > >>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> > > >>>> common
> > > >>>>>>>> built-in
> > > >>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> > > >>> every
> > > >>>>>>>>>> CatalogSotre
> > > >>>>>>>>>>>>>>>>> implementation has to take care of it.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > >>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi Jing,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks for your reply.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > > >>>>> suggest
> > > >>>>>>> you
> > > >>>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>> walk
> > > >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> > > >>> the
> > > >>>>> same
> > > >>>>>>>> time, I
> > > >>>>>>>>>>>>> greatly
> > > >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> > > >>>> thoroughly
> > > >>>>>>>> checked
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> entire
> > > >>>>>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> > > >>> FLIP
> > > >>>>>>> design,
> > > >>>>>>>>>>>>>>> CatalogManager
> > > >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> > > >>> being
> > > >>>>>>>> configured.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> > > >>> was
> > > >>>> not
> > > >>>>>>>> necessary
> > > >>>>>>>>>>>>> because
> > > >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> > > >>>> catalogs
> > > >>>>>> to
> > > >>>>>>>> store
> > > >>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>> instances.
> > > >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> > > >>>>>>> Therefore, I
> > > >>>>>>>>>>>> modified
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> part
> > > >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> > > >>> Catalog>
> > > >>>>>>> catalogs
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>> CatalogManager.
> > > >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> > > >>> serve
> > > >>>> as
> > > >>>>>> the
> > > >>>>>>>>>> default
> > > >>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> > > >>>>>>>> functionality of
> > > >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> > > >>>>>>>>>>>>>>>>>> The previous plan that kept
> > > >>>>>>>> Map<String,Catalog>catalogs has
> > > >>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>> moved
> > > >>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > >>>>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There are still many places contain
> > > >>>>> inconsistent
> > > >>>>>>>> content,
> > > >>>>>>>>>>>> e.g.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> > > >>> used.
> > > >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> > > >>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> > > >>> not
> > > >>>>> tell
> > > >>>>>>>> what the
> > > >>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> doing.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > > >>>>> suggest
> > > >>>>>>> you
> > > >>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>> walk
> > > >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> > > >>> issues.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> > > >>> mean
> > > >>>>> that
> > > >>>>>>> you
> > > >>>>>>>> will
> > > >>>>>>>>>>>> build
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> > > >>> is a
> > > >>>>> very
> > > >>>>>>>>>> different
> > > >>>>>>>>>>>>>> design
> > > >>>>>>>>>>>>>>>>>> concept
> > > >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> > > >>> I am
> > > >>>>> not
> > > >>>>>>>>>> mistaken,
> > > >>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> > > >>>> work
> > > >>>>>>>> without
> > > >>>>>>>>>>>> Optional
> > > >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> > > >>>>> reason
> > > >>>>>>>> why I
> > > >>>>>>>>>>>>> mentioned
> > > >>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> > > >>>>> Optional
> > > >>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>> correct.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > >>>>>>>>>>>>> jinfeng1995@gmail.com>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi Jing
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> > > >>>> FLIP
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> > > >>>> review. I
> > > >>>>>>> have
> > > >>>>>>>>>>> already
> > > >>>>>>>>>>>>> made
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> relevant changes.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> > > >>> the
> > > >>>>>>>>>> multi-instance
> > > >>>>>>>>>>>>>>>> scenarios, I
> > > >>>>>>>>>>>>>>>>>> am
> > > >>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> > > >>> multiple
> > > >>>>>>>> scenarios
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> current
> > > >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> > > >>>>> remove
> > > >>>>>>>>>>> 'Map<String,
> > > >>>>>>>>>>>>>>>> Catalog>'
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> > > >>>>>>>>>> InMemoryCatalogStore.
> > > >>>>>>>>>>>> This
> > > >>>>>>>>>>>>>>> way,
> > > >>>>>>>>>>>>>>>>>> users
> > > >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> > > >>>>> achieve
> > > >>>>>>>>>>>>> multi-instance
> > > >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> > > >>>> think?
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > >>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> > > >>> are
> > > >>>>>> still
> > > >>>>>>>> some
> > > >>>>>>>>>>> NIT
> > > >>>>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>>>>>> like:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> > > >>>> has
> > > >>>>>> been
> > > >>>>>>>> used
> > > >>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> > > >>> example.
> > > >>>> It
> > > >>>>>>>> should be
> > > >>>>>>>>>>>> fine
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> use
> > > >>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> > > >>> after
> > > >>>> you
> > > >>>>>>>> submit
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> PR.
> > > >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> > > >>>> somewhere
> > > >>>>> in
> > > >>>>>>> the
> > > >>>>>>>>>>>> rejected
> > > >>>>>>>>>>>>>>>> section
> > > >>>>>>>>>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> > > >>> it
> > > >>>>>>>> consistent.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> > > >>>>>>>> multi-instance
> > > >>>>>>>>>>>>>>> scenarios,
> > > >>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>> am
> > > >>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> > > >>> behaviour.
> > > >>>> If
> > > >>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> > > >>>> once
> > > >>>>>> the
> > > >>>>>>>>>> related
> > > >>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> has
> > > >>>>>>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> > > >>> another
> > > >>>>>>>> instance.
> > > >>>>>>>>>> Did
> > > >>>>>>>>>>> I
> > > >>>>>>>>>>>>> miss
> > > >>>>>>>>>>>>>>>>>>> something?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> > > >>> Jin
> > > >>>> <
> > > >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
> > > >>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> @Jing
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> > > >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> > > >>> name, I
> > > >>>>>> have
> > > >>>>>>>>>> updated
> > > >>>>>>>>>>> it
> > > >>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> documentation.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> > > >>> should
> > > >>>> be
> > > >>>>>>>>>> implemented
> > > >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> > > >>>>>> alternative
> > > >>>>>>> 1
> > > >>>>>>>>>> turns
> > > >>>>>>>>>>>> out
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> > > >>> this
> > > >>>>> part
> > > >>>>>>> of
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> content.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> > > >>>> forward,
> > > >>>>>> the
> > > >>>>>>>>>> option 2
> > > >>>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> > > >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> > > >>>>> offline,
> > > >>>>>>> we
> > > >>>>>>>>>> think
> > > >>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>> 2
> > > >>>>>>>>>>>>>>>>>>>>> might
> > > >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> > > >>>>>>>> inconsistency
> > > >>>>>>>>>>>> issues.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> > > >>>>> descriptions
> > > >>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> content.
> > > >>>>>>>>>>>>>>>> Would
> > > >>>>>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> > > >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> > > >>>> document
> > > >>>>>> and
> > > >>>>>>>>>>>> appreciate
> > > >>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>>> suggestions.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> > > >>> in
> > > >>>>>>>> `Rejected
> > > >>>>>>>>>>>>>>> Alternatives`
> > > >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> > > >>> `Rejected
> > > >>>>>>>>>> Alternatives`.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best
> > > >>>>>>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> > > >>> Shammon
> > > >>>>> FY
> > > >>>>>> <
> > > >>>>>>>>>>>>>>> zjureel@gmail.com>
> > > >>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> > > >>>>>> `Proposed
> > > >>>>>>>>>>> Changes`,
> > > >>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>> put
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> > > >>>>>>>> Alternatives`? I
> > > >>>>>>>>>>> think
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>>> help
> > > >>>>>>>>>>>>>>>>>>>> us
> > > >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> > > >>> Jing
> > > >>>> Ge
> > > >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> > > >>> am
> > > >>>>>> still
> > > >>>>>>>>>> confused
> > > >>>>>>>>>>>>> after
> > > >>>>>>>>>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>>>>>>>>>> reading
> > > >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> > > >>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> > > >>>>> registerCatalog()
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> addCatalog()
> > > >>>>>>>>>>>>>>>> have
> > > >>>>>>>>>>>>>>>>>> no
> > > >>>>>>>>>>>>>>>>>>>> big
> > > >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> > > >>>> One
> > > >>>>> of
> > > >>>>>>>> them is
> > > >>>>>>>>>>>>>>> responsible
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>> data
> > > >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> > > >>>>>> persistCatalog()?
> > > >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> > > >>>> Map<String,
> > > >>>>>>>> Catalog>
> > > >>>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>> as a
> > > >>>>>>>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> > > >>> data
> > > >>>>>>>> persistence.
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>> suggest
> > > >>>>>>>>>>>>>>>>>>>>>> describing
> > > >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> > > >>>>> clearly
> > > >>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>> Some
> > > >>>>>>>>>>>>>>>>> common
> > > >>>>>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> > > >>>> i.e.
> > > >>>>>> data
> > > >>>>>>>> in
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> store
> > > >>>>>>>>>>>>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> > > >>>> instance
> > > >>>>>>>> should be
> > > >>>>>>>>>>>> found
> > > >>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> store
> > > >>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> > > >>>>>> initialized
> > > >>>>>>>> or it
> > > >>>>>>>>>>> will
> > > >>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>> lazy
> > > >>>>>>>>>>>>>>>>>>>>>> initialized)
> > > >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> > > >>>>>>> consistency
> > > >>>>>>>> will
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>>> taken
> > > >>>>>>>>>>>>>>>> care
> > > >>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>> while
> > > >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> > > >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> > > >>>>>> forward,
> > > >>>>>>>> the
> > > >>>>>>>>>>>> option
> > > >>>>>>>>>>>>> 2
> > > >>>>>>>>>>>>>>>>> solution
> > > >>>>>>>>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> > > >>>>>> because,
> > > >>>>>>>> afaiu,
> > > >>>>>>>>>>>>> issues
> > > >>>>>>>>>>>>>>>>>> mentioned
> > > >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> > > >>>>> solved
> > > >>>>>>>> yet. Do
> > > >>>>>>>>>>> you
> > > >>>>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>>>> want
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> propose
> > > >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> > > >>>> suggestions
> > > >>>>>> for
> > > >>>>>>>> both
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>>> them?
> > > >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> > > >>>> there
> > > >>>>>> are
> > > >>>>>>>> some
> > > >>>>>>>>>>>>>>> inconsistent
> > > >>>>>>>>>>>>>>>>>>>>> descriptions
> > > >>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> > > >>>> clean
> > > >>>>>>> them
> > > >>>>>>>> up?
> > > >>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Jing
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> > > >>> Feng
> > > >>>>>> Jin <
> > > >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> > > >>> and
> > > >>>> I
> > > >>>>>>>> completely
> > > >>>>>>>>>>>> agree
> > > >>>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>> point
> > > >>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>> view.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> > > >>>>>> complicated
> > > >>>>>>>> the
> > > >>>>>>>>>>> whole
> > > >>>>>>>>>>>>>> issue.
> > > >>>>>>>>>>>>>>>>> First
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> > > >>>> persistence
> > > >>>>>> of
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> Catalog's
> > > >>>>>>>>>>>>>>>>>>>> Configuration.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> > > >>>>>> persistence
> > > >>>>>>>> for
> > > >>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>> Configuration,
> > > >>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> > > >>> to
> > > >>>> the
> > > >>>>>>>> Catalog
> > > >>>>>>>>>>>>>> interface.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> > > >>> Catalog
> > > >>>>>>>> instance
> > > >>>>>>>>>> to a
> > > >>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>> String>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> > > >>>>>>>> implementation
> > > >>>>>>>>>> will
> > > >>>>>>>>>>>>> throw
> > > >>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>> exception.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> > > >>>>>>>>>>>>>>>>>>>>>>>>> /**
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> > > >>> the
> > > >>>>>>>> properties
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>> object.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> *
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> > > >>> the
> > > >>>>>>>> properties
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>> object
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> > > >>>>>>>> UnsupportedOperationException if
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> implementing
> > > >>>>>>>>>>>>>>>>>>>>> class
> > > >>>>>>>>>>>>>>>>>>>>>>> does
> > > >>>>>>>>>>>>>>>>>>>>>>>>> not override
> > > >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> > > >>> implementation
> > > >>>> of
> > > >>>>>> this
> > > >>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> */
> > > >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> > > >>>>>>>> toProperties()
> > > >>>>>>>>>> {
> > > >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> > > >>>>>>>>>>>> UnsupportedOperationException("Please
> > > >>>>>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> > > >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> > > >>>> follows:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> > > >>> configured a
> > > >>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> > > >>>>>>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> > > >>> registering a
> > > >>>>>>> Catalog
> > > >>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > > >>>> catalogName,
> > > >>>>>>>> Catalog
> > > >>>>>>>>>>>>> catalog).
> > > >>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> > > >>>> Map<String,
> > > >>>>>>>> String>
> > > >>>>>>>>>>>>> properties
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>> saved
> > > >>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> > > >>>>>> instances
> > > >>>>>>>> do
> > > >>>>>>>>>> not
> > > >>>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> method,
> > > >>>>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> > > >>>>>>>> CatalogStore,
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> toConfiguration()
> > > >>>>>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> > > >>>>>>> consistency
> > > >>>>>>>> with
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> original
> > > >>>>>>>>>>>>>>>>>>>> process.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> > > >>>>>> Catalog>
> > > >>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> > > >>>>>> conflicts
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> > > >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> > > >>>>> itself
> > > >>>>>>> and
> > > >>>>>>>>>>> provide
> > > >>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>> dedicated
> > > >>>>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> > > >>> loading,
> > > >>>>> such
> > > >>>>>>> as
> > > >>>>>>>>>>>>>>>>> ConfigurationCatalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> public class
> > > >>> ConfigurationCatalog
> > > >>>>>>>> implements
> > > >>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>> {
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>> ConfigurationCatalog(Map<String,
> > > >>>>>>> String>
> > > >>>>>>>>>>>>> properties)
> > > >>>>>>>>>>>>>> {
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> > > >>> the
> > > >>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> > > >>> 10:03 AM
> > > >>>>>> Shammon
> > > >>>>>>>> FY <
> > > >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> > > >>> the
> > > >>>>>>>> functionality
> > > >>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> > > >>>>>>>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> > > >>>> multiple
> > > >>>>>>>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > > >>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> > > >>>>>> implementation
> > > >>>>>>> in
> > > >>>>>>>>>> Trino
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> same,
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> > > >>> only
> > > >>>>> has
> > > >>>>>>> one
> > > >>>>>>>>>>>>>>> `createCatalog`,
> > > >>>>>>>>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>> save
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> > > >>>>>> `CatalogStore`.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> > > >>>>>> another
> > > >>>>>>>>>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> > > >>> `Map<String,
> > > >>>>>>> Catalog>
> > > >>>>>>>>>>>> catalogs`
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> > > >>>> mentioned
> > > >>>>>>>> before,
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> `Map<String,
> > > >>>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> > > >>>>>>>> `CatalogManager`.
> > > >>>>>>>>>> How
> > > >>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>> saving
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> > > >>> catalogs`
> > > >>>>> and
> > > >>>>>>>>>>>> `CatalogStore
> > > >>>>>>>>>>>>>>>>>>> catalogStore`
> > > >>>>>>>>>>>>>>>>>>>>>>>> together
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> > > >>> added
> > > >>>> in
> > > >>>>>>>>>>>>> `CatalogManager`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > > >>> 5:22 PM
> > > >>>>> Feng
> > > >>>>>>>> Jin <
> > > >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> > > >>> I
> > > >>>> am
> > > >>>>>> very
> > > >>>>>>>> sorry
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> > > >>>>>>>>>>>>>>>>>>>>>>>>> caused
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> > > >>>>> original
> > > >>>>>>>>>>> discussion.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > > >>>>>> pre-discussion
> > > >>>>>>>> [1]
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> > > >>> such a
> > > >>>>>>>> discussion
> > > >>>>>>>>>>>> before.
> > > >>>>>>>>>>>>>>>>> However,
> > > >>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> designing
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> > > >>> that
> > > >>>>> the
> > > >>>>>>>> logic of
> > > >>>>>>>>>>>>>>>>> CatalogManager
> > > >>>>>>>>>>>>>>>>>>>> itself
> > > >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> > > >>>> cannot
> > > >>>>>>> only
> > > >>>>>>>>>>> persist
> > > >>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>> instances
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> > > >>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> might
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> > > >>>> Catalogs,
> > > >>>>>>>> because
> > > >>>>>>>>>> in
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> end
> > > >>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>>>>>>>> need
> > > >>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> save
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> > > >>>>> corresponding
> > > >>>>>> to
> > > >>>>>>>> the
> > > >>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>> instances.
> > > >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> > > >>>>>>> CatalogStore
> > > >>>>>>>>>>>> interface
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> > > >>> this
> > > >>>>>> part, I
> > > >>>>>>>> also
> > > >>>>>>>>>>> took
> > > >>>>>>>>>>>>>>>>> inspiration
> > > >>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > > >>>> handling
> > > >>>>>>>> catalog
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>>>>>>>> looks a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> little
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> > > >>>> suggestion. I
> > > >>>>>>> will
> > > >>>>>>>>>>> provide
> > > >>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>> detailed
> > > >>>>>>>>>>>>>>>>>>>>>> description
> > > >>>>>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> > > >>> part,
> > > >>>>> and
> > > >>>>>>> add
> > > >>>>>>>> it
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> > > >>>>>>>> `addCatalog`
> > > >>>>>>>>>>> still
> > > >>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>> confused
> > > >>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> > > >>>>>> catalog
> > > >>>>>>> to
> > > >>>>>>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>>>>>>>> renaming
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> > > >>> very
> > > >>>>>>> personal
> > > >>>>>>>>>>>> opinion,
> > > >>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>> decide
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> > > >>>>> intuitive
> > > >>>>>>> to
> > > >>>>>>>> me.
> > > >>>>>>>>>> I
> > > >>>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>>> see
> > > >>>>>>>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>>>>>>> problem
> > > >>>>>>>>>>>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> > > >>>>> statement,
> > > >>>>>>>> which
> > > >>>>>>>>>>>> method
> > > >>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>> called?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > > >>>>>> `addCatalog`?
> > > >>>>>>>> I'm
> > > >>>>>>>>>>>>> wondering
> > > >>>>>>>>>>>>>>>>> whether
> > > >>>>>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>> add a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> > > >>>>>> stement.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> > > >>>> original
> > > >>>>>>>> design
> > > >>>>>>>>>> was
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>> add
> > > >>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>> directly
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> > > >>> would
> > > >>>>>>> disrupt
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> existing
> > > >>>>>>>>>>>>>>>>> logic.
> > > >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> > > >>>>>>>> configuration to
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> > > >>> the
> > > >>>>> same
> > > >>>>>>> time
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> > > >>> neccessary
> > > >>>> to
> > > >>>>>>>> provide a
> > > >>>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>>>>> implmentation
> > > >>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > > >>>>> `CatalogStoreFactory`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> > > >>>>>> otherwise
> > > >>>>>>> we
> > > >>>>>>>>>> would
> > > >>>>>>>>>>>>> need
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> introduce
> > > >>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> > > >>>> configuration
> > > >>>>>> for
> > > >>>>>>>> lazy
> > > >>>>>>>>>>>>> loading.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > > >>>>>>> registration
> > > >>>>>>>> for
> > > >>>>>>>>>>>>> catalog.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> > > >>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> > > >>>>>> interface
> > > >>>>>>>>>> because
> > > >>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> > > >>>>>>>> initialization
> > > >>>>>>>>>>> for
> > > >>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> > > >>>>>>>> inaccurate. It
> > > >>>>>>>>>>>>> should
> > > >>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> lazy
> > > >>>>>>>>>>>>>>>>>>>>>>> registration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> > > >>>>>>>> registration. I
> > > >>>>>>>>>>> have
> > > >>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>> updated
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> title
> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > > >>>> 4:27 PM
> > > >>>>>>>> Shammon
> > > >>>>>>>>>> FY <
> > > >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> > > >>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> > > >>>>> method
> > > >>>>>>>> named
> > > >>>>>>>>>>>>>> `addCatalog`
> > > >>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> > > >>> the
> > > >>>>>> memory
> > > >>>>>>>>>> catalog
> > > >>>>>>>>>>>> is a
> > > >>>>>>>>>>>>>>>>> `cache`,
> > > >>>>>>>>>>>>>>>>>> I
> > > >>>>>>>>>>>>>>>>>>>>> think
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> > > >>>>>> exposed
> > > >>>>>>> to
> > > >>>>>>>>>>> users.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > > >>>>>> pre-discussion
> > > >>>>>>>> [1]
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>> Please
> > > >>>>>>>>>>>>>>>>>>>>>>> correct
> > > >>>>>>>>>>>>>>>>>>>>>>>>> me
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> > > >>>>> conclusion
> > > >>>>>>> of
> > > >>>>>>>> that
> > > >>>>>>>>>>>>>>> discussion
> > > >>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>> use
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> > > >>>>>> interface
> > > >>>>>>>> and
> > > >>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> different
> > > >>>>>>>>>>>>>>>>>>>>>>>>> stores
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> > > >>>> external
> > > >>>>>>>> system.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> > > >>>>> between
> > > >>>>>>> the
> > > >>>>>>>>>>> current
> > > >>>>>>>>>>>>> FLIP
> > > >>>>>>>>>>>>>>>>> design
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> > > >>> of
> > > >>>>> the
> > > >>>>>>>>>> discussion
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> thread
> > > >>>>>>>>>>>>>>>>>> [1] ?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > > >>>>> 3:41 PM
> > > >>>>>>>> yuxia <
> > > >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> > > >>> driving
> > > >>>>> this
> > > >>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > > >>>>> handling
> > > >>>>>>>> catalog
> > > >>>>>>>>>>> with
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>>>>>>>>> looks a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> little
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> > > >>>>> it'll
> > > >>>>>> be
> > > >>>>>>>>>> better
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> explain
> > > >>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> java
> > > >>>>>>>>>>>>>>>>>>>>>>>>> doc
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> > > >>> give a
> > > >>>>>> brief
> > > >>>>>>>>>> example
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> FLIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> > > >>> name
> > > >>>>>>>> `addCatalog`
> > > >>>>>>>>>>>> still
> > > >>>>>>>>>>>>>>> looks
> > > >>>>>>>>>>>>>>>>>>> confused
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> > > >>>> storing
> > > >>>>>>>> catalog to
> > > >>>>>>>>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> > > >>> It's
> > > >>>>> very
> > > >>>>>>>> personal
> > > >>>>>>>>>>>>>> opinion,
> > > >>>>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>> decide
> > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> > > >>>>>>> statement,
> > > >>>>>>>>>> which
> > > >>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>> called?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > > >>>>>>>> `addCatalog`?
> > > >>>>>>>>>> I'm
> > > >>>>>>>>>>>>>>> wondering
> > > >>>>>>>>>>>>>>>>>>> whether
> > > >>>>>>>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> > > >>> SQL
> > > >>>>>>>> stement.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> > > >>>> neccessary
> > > >>>>> to
> > > >>>>>>>>>> provide a
> > > >>>>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>>>>>> implmentation
> > > >>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > > >>>>>>> `CatalogStoreFactory`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > > >>>>>>>> registration
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> catalog.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> > > >>> catalog
> > > >>>>>> with
> > > >>>>>>>> CREATE
> > > >>>>>>>>>>>>>> CATALOG,
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > > >>>> used
> > > >>>>>> by
> > > >>>>>>>>>> default.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> > > >>>>> registration
> > > >>>>>> is
> > > >>>>>>>> the
> > > >>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>> behavior,
> > > >>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> way
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> > > >>> to
> > > >>>>>>>> synchronous
> > > >>>>>>>>>>>>>>> registration
> > > >>>>>>>>>>>>>>>>> just
> > > >>>>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>>>>> before?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> > > >>>>> `addCatalog`
> > > >>>>>>> and
> > > >>>>>>>>>>>>>>>> `registerCatalog`
> > > >>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> > > >>>>>>>> registration,
> > > >>>>>>>>>> it
> > > >>>>>>>>>>>> may
> > > >>>>>>>>>>>>>> well
> > > >>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> > > >>>> but
> > > >>>>>> then
> > > >>>>>>>> the
> > > >>>>>>>>>>>>> following
> > > >>>>>>>>>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>>>>>>>> TABLE
> > > >>>>>>>>>>>>>>>>>>>>>>>>> statement
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> > > >>> fail
> > > >>>>> to
> > > >>>>>>>> open.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> > > >>>> change
> > > >>>>>>> which
> > > >>>>>>>>>>> should
> > > >>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>> highlighted
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> > > >>> part.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> > > >>>> asynchronous
> > > >>>>>>>>>>>> registration, I
> > > >>>>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> expect
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> > > >>>> open
> > > >>>>>> or
> > > >>>>>>>>>> register
> > > >>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> background,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> > > >>> comments,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> > > >>>> will
> > > >>>>> be
> > > >>>>>>>>>>> initialized
> > > >>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>> has
> > > >>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> > > >>>> initialization
> > > >>>>>>>> process
> > > >>>>>>>>>>> fails,
> > > >>>>>>>>>>>>>> these
> > > >>>>>>>>>>>>>>>>>>>> statements
> > > >>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> > > >>> successfully."
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> > > >>> lazy
> > > >>>>>>>>>> initialization
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> > > >>>>>>>>>> jinfeng1995@gmail.com
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> > > >>>>>>> dev@flink.apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> > > >>> 03日
> > > >>>> 下午
> > > >>>>>>>> 3:27:45
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> > > >>>> 295:
> > > >>>>>>>> Support
> > > >>>>>>>>>>>>>> persistence
> > > >>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> > > >>>>> registration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> > > >>>> all
> > > >>>>>> for
> > > >>>>>>>> your
> > > >>>>>>>>>>>>> interest
> > > >>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> > > >>> catalog
> > > >>>>>> with
> > > >>>>>>>> the
> > > >>>>>>>>>> same
> > > >>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>> both
> > > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> > > >>>> crucial
> > > >>>>>>>> point.
> > > >>>>>>>>>>> Based
> > > >>>>>>>>>>>> on
> > > >>>>>>>>>>>>>> my
> > > >>>>>>>>>>>>>>>>>> current
> > > >>>>>>>>>>>>>>>>>>>>>> personal
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> > > >>>>>> Map<String,
> > > >>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>> serve
> > > >>>>>>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> > > >>> catalogs
> > > >>>>> and
> > > >>>>>>>> have
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> highest
> > > >>>>>>>>>>>>>>>>>>> priority.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> > > >>> methods
> > > >>>>> that
> > > >>>>>>> can
> > > >>>>>>>>>> affect
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> > > >>> registerCatalog(String
> > > >>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> > > >>>>> catalog
> > > >>>>>>>> instance
> > > >>>>>>>>>>>> into
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > > >>>> unregisterCatalog(String
> > > >>>>>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > > >>> the
> > > >>>>>> catalog
> > > >>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>> corresponding
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> > > >>>>> Catalog>
> > > >>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> > > >>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> > > >>>> retrieves
> > > >>>>>> the
> > > >>>>>>>>>>>>> corresponding
> > > >>>>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> > > >>> Catalog>
> > > >>>>>>>> catalogs. If
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>>>> does
> > > >>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>> exist,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> > > >>>> corresponding
> > > >>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> > > >>> puts
> > > >>>>> the
> > > >>>>>>>>>>> initialized
> > > >>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>> into
> > > >>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > > >>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> > > >>> methods
> > > >>>>>> only
> > > >>>>>>>> modify
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> > > >>>>>>> catalogName,
> > > >>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>> String>
> > > >>>>>>>>>>>>>>>>>>>>>>> properties)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> > > >>>>>>> properties
> > > >>>>>>>> to
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> catalogStore
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>> checks
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> > > >>>>>> catalogName
> > > >>>>>>>> with
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>> name.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > > >>> removeCatalog(String
> > > >>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > > >>> the
> > > >>>>>>> specified
> > > >>>>>>>>>>>>>> configuration
> > > >>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> specified
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> > > >>>>>>> catalogStore.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> > > >>>> possible
> > > >>>>>>>> conflict
> > > >>>>>>>>>>>>>> scenarios:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> > > >>> corresponding
> > > >>>>>>>> catalogName
> > > >>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> > > >>> in
> > > >>>> the
> > > >>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>> Catalog>,
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > > >>>>>>>> catalogName,
> > > >>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> > > >>> directly
> > > >>>>> saved
> > > >>>>>>> to
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> > > >>> corresponding
> > > >>>>>>>> catalogName
> > > >>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> both
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> > > >>>>>>> Map<String,
> > > >>>>>>>>>>>> Catalog>,
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> > > >>>>> catalog)
> > > >>>>>>>> method
> > > >>>>>>>>>>> will
> > > >>>>>>>>>>>>>> fail.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> > > >>> corresponding
> > > >>>>>>>> catalogName
> > > >>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> exists
> > > >>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> > > >>> the
> > > >>>>>>>>>>>> addCatalog(String
> > > >>>>>>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> > > >>>> method
> > > >>>>>> can
> > > >>>>>>>>>> directly
> > > >>>>>>>>>>>>> save
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> properties
> > > >>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> > > >>>>>>>>>> getCatalog(String
> > > >>>>>>>>>>>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> > > >>>> for
> > > >>>>>>>>>>> initialization
> > > >>>>>>>>>>>>>>> because
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> > > >>> already
> > > >>>>>> exists
> > > >>>>>>>> in
> > > >>>>>>>>>>>> catalogs
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> > > >>>>>>>>>>>>> unregisterCatalog(String
> > > >>>>>>>>>>>>>>>>>>>> catalogName)
> > > >>>>>>>>>>>>>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> > > >>>>>>>> corresponding to
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>> original
> > > >>>>>>>>>>>>>>>>>>>>> catalogName
> > > >>>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> > > >>> confuse
> > > >>>>>> users
> > > >>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> > > >>>>> catalog)`
> > > >>>>>>> in
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>> `Map<String,
> > > >>>>>>>>>>>>>>>>>>>> Catalog>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>> `registerCatalog(String
> > > >>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>> String>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> > > >>>> lead
> > > >>>>>> to
> > > >>>>>>>>>>>> confusion. I
> > > >>>>>>>>>>>>>>>> suggest
> > > >>>>>>>>>>>>>>>>>>>>> changing
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> > > >>> to
> > > >>>>>>>>>>> addCatalog(String
> > > >>>>>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> > > >>> as
> > > >>>>>>>> previously
> > > >>>>>>>>>>>>> mentioned
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> > > >>>>> `registerCatalog(String
> > > >>>>>>>>>>>>>>> catalogName,Catalog
> > > >>>>>>>>>>>>>>>>>>>> catalog,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> > > >>> method
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> > > >>>> already
> > > >>>>> an
> > > >>>>>>>>>> instance,
> > > >>>>>>>>>>>>>> adding
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> > > >>>>>>>>>> registerCatalog(String
> > > >>>>>>>>>>>>>>>>> catalogName,
> > > >>>>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> > > >>>> necessarily
> > > >>>>>>>> result in
> > > >>>>>>>>>>> lazy
> > > >>>>>>>>>>>>>>>>>>> initialization.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> > > >>>> about
> > > >>>>>>>>>> encryption
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> > > >>>>>>> necessary,
> > > >>>>>>>> but
> > > >>>>>>>>>>>>> perhaps
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> encryption
> > > >>>>>>>>>>>>>>>>>>>>>>> logic
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> > > >>> in
> > > >>>>> the
> > > >>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>> when
> > > >>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> data
> > > >>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>> actually
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> > > >>> the
> > > >>>>>>>>>>> FileCatalogStore
> > > >>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>> encrypt
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> data
> > > >>>>>>>>>>>>>>>>>>>>>>>>> when
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> > > >>>>> while
> > > >>>>>>> the
> > > >>>>>>>>>>>>>>>> MemoryCatalogStore
> > > >>>>>>>>>>>>>>>>>>> does
> > > >>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>> require
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> > > >>> the
> > > >>>>>>>>>>>>>> `MemoryCatalogStore`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> > > >>> necessary
> > > >>>> to
> > > >>>>>>> have a
> > > >>>>>>>>>>>>>>>>> MemoryCatalogStore
> > > >>>>>>>>>>>>>>>>>> as
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> default
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> > > >>>> saves
> > > >>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>> configurations
> > > >>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> memory.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> > > >>> to
> > > >>>>>>>> implement
> > > >>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>> loading
> > > >>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> > > >>> introduce
> > > >>>>>>>> additional
> > > >>>>>>>>>>> cache.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> > > >>>>>>>> registration
> > > >>>>>>>>>>>> failed?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> > > >>> critical
> > > >>>>>>> concern.
> > > >>>>>>>>>> When
> > > >>>>>>>>>>>>>>> executing
> > > >>>>>>>>>>>>>>>>> DDL,
> > > >>>>>>>>>>>>>>>>>>>> DQL,
> > > >>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>> DML
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> > > >>>> reference a
> > > >>>>>>>> specific
> > > >>>>>>>>>>>>> Catalog,
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> Catalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> > > >>> if it
> > > >>>>> has
> > > >>>>>>> not
> > > >>>>>>>>>> been
> > > >>>>>>>>>>>>>>>> initialized
> > > >>>>>>>>>>>>>>>>>> yet.
> > > >>>>>>>>>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> > > >>>>> fails,
> > > >>>>>>>> these
> > > >>>>>>>>>>>>>> statements
> > > >>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>> executed
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> > > >>>>>> Catalog>
> > > >>>>>>>>>> catalogs
> > > >>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> > > >>>> have
> > > >>>>>> the
> > > >>>>>>>> same
> > > >>>>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>>>> name,
> > > >>>>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>>>> correponding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > > >>>>> instances.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> > > >>>> resolved
> > > >>>>> by
> > > >>>>>>>>>> referring
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> previous
> > > >>>>>>>>>>>>>>>>>>>>>>>> responses.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> > > >>> the
> > > >>>>>>> Catalog
> > > >>>>>>>>>> that
> > > >>>>>>>>>>>> has
> > > >>>>>>>>>>>>>> been
> > > >>>>>>>>>>>>>>>>> most
> > > >>>>>>>>>>>>>>>>>>>>> recently
> > > >>>>>>>>>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> > > >>>> for
> > > >>>>>>>> subsequent
> > > >>>>>>>>>>>> use.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> > > >>>>>> 10:58 PM
> > > >>>>>>>>>> Xianxun
> > > >>>>>>>>>>>> Ye
> > > >>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> > > >>> this
> > > >>>>>> Flip,
> > > >>>>>>>> I do
> > > >>>>>>>>>>>>> believe
> > > >>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>> Flip
> > > >>>>>>>>>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> > > >>>> work
> > > >>>>>> for
> > > >>>>>>>> also
> > > >>>>>>>>>>>>> manages a
> > > >>>>>>>>>>>>>>> lot
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>> catalogs,
> > > >>>>>>>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> > > >>>> becomes
> > > >>>>>>> slow
> > > >>>>>>>>>>> because
> > > >>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>> loading a
> > > >>>>>>>>>>>>>>>>>>>>> number
> > > >>>>>>>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> > > >>> catalogs in
> > > >>>>> the
> > > >>>>>>>> user's
> > > >>>>>>>>>>>> Flink
> > > >>>>>>>>>>>>>> SQL
> > > >>>>>>>>>>>>>>>>>> through
> > > >>>>>>>>>>>>>>>>>>>>>> regular
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> > > >>>>> catalogs
> > > >>>>>>> to
> > > >>>>>>>>>>> improve
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> speed
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>> task
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> > > >>> flip-295, I
> > > >>>>>> have
> > > >>>>>>>> some
> > > >>>>>>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> > > >>>>> catalog
> > > >>>>>>> with
> > > >>>>>>>>>>> CREATE
> > > >>>>>>>>>>>>>>> CATALOG,
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > > >>>> used
> > > >>>>>> by
> > > >>>>>>>>>> default.
> > > >>>>>>>>>>>>> What
> > > >>>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> > > >>>>> implement
> > > >>>>>>>>>>> asynchronous
> > > >>>>>>>>>>>>>>>>>> registration?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> > > >>> same
> > > >>>>>>>> question
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>>>> Shammon’s
> > > >>>>>>>>>>>>>>>>>> first
> > > >>>>>>>>>>>>>>>>>>>>>>> comment.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > > >>>>> catalogs
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> CatalogStore
> > > >>>>>>>>>>>>>>>>>>>> catalogstore
> > > >>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> > > >>> have
> > > >>>> the
> > > >>>>>>> same
> > > >>>>>>>>>>> catalog
> > > >>>>>>>>>>>>>> name,
> > > >>>>>>>>>>>>>>>> but
> > > >>>>>>>>>>>>>>>>>>>>>> correponding
> > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > > >>>>> instances.
> > > >>>>>>> For
> > > >>>>>>>>>>> example:
> > > >>>>>>>>>>>>>>>>> catalog1->
> > > >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> > > >>>>>>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> > > >>> catalog1 ->
> > > >>>>>>>> hiveCatalog
> > > >>>>>>>>>>> in
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> catalogstore.
> > > >>>>>>>>>>>>>>>>>>>>>>> Would
> > > >>>>>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> > > >>> should
> > > >>>>>> define
> > > >>>>>>>>>>> `catalogs`
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> > > >>>>>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> > > >>> 11:13，Hang
> > > >>>>>> Ruan
> > > >>>>>>> <
> > > >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> 写道：
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Feng.
I reviewed this FLIP again and found some little places that may need to
optimize.
1. `CatalogDescriptor` may need a private constructor.
2. The method `initCatalog` in `CatalogManager` is not introduced.
Best,
Hang
Feng Jin <ji...@gmail.com> 于2023年6月6日周二 21:17写道：
> Hi Leonard,
>
> Thanks for your reply.
>
> > 1. a How to construct a CatalogDescriptor ?
>
> I think it would be helpful to add a method for constructing a
> CatalogDescriptor, as you mentioned in 1.c. I will update the documentation
> later.
>
> > 1.b How to visit the fields ? Could we use Configuration instead of
> Map<String, String> ?
>
> I believe that the use of Map<String, String> options is only intended for
> creating a catalog and not for accessing internal parameters.
>
> Since all of the relevant parameters for CREATE CATALOG are also stored in
> Map<String, String> options, my understanding is that using Map<String,
> String> options should suffice.
>
> Here is the implementation of execute CREATE CATALOG statement.
> ```java
> private TableResultInternal createCatalog(CreateCatalogOperation operation)
> {
> String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
> try {
> String catalogName = operation.getCatalogName();
> Map<String, String> properties = operation.getProperties();
>
> Catalog catalog =
> FactoryUtil.createCatalog(
> catalogName,
> properties,
> tableConfig,
> resourceManager.getUserClassLoader());
> catalogManager.registerCatalog(catalogName, catalog);
>
> return TableResultImpl.TABLE_RESULT_OK;
> } catch (CatalogException e) {
> throw new ValidationException(exMsg, e);
> }
> }
> ```
>
>
> > 2. Do we have plan to offer a default CatalogStore if user didn’t config
> this?
>
> Yes, the in-memory catalogStore will be used as the default CatalogStore
> even if the user has not configured one
>
>
> Best,
> Feng
>
>
> On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
>
> > Hi, Feng
> >
> > Sorry for reply late, but I’ve some comments about the FLIP
> >
> >
> > 1. The introduced Public class CatalogDescriptor seems missed some
> > necessary component
> > a) How to construct a CatalogDescriptor ?
> > b) How to visit the fields ? Could we use Configuration instead of
> > Map<String, String> ?
> > c) Could we offer a built-in factory method to build a
> CatalogDescriptor
> > like
> > public static CatalogDescriptor of(String catalogName, Configuration
> > configuration)
> >
> > 2. The FLIP said “By default, there are two built-in CatalogStores
> > available: the In-Memory CatalogStore and the File CatalogStore” ,
> > Do we have plan to offer a default CatalogStore if user didn’t config
> > this? IIUC, users can obtains the benefits from lazy catalog
> > initialization If
> > we have a default catalogstore even it is in-memory.
> >
> > Best,
> > Leonard
> >
> >
> >
> > > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > Thank you everyone for your valuable input. If there are no further
> > > questions or concerns about the FLIP[1], I would like to start voting
> > > tomorrow (6/7).
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
> > >
> > >> Hi Samrat,
> > >>
> > >> Thanks for your advice.
> > >>
> > >>> 1. The createCatalog method does not mention any exceptions being
> > >> thrown.
> > >>
> > >> CreateCatalog will throw CatalogException like registerCatalog. As
> > >> CatalogException is a RuntimeException,
> > >> there is no explicit declaration of throwing Exceptions in
> > CatalogManager
> > >> and TableEnvironment.
> > >> To avoid misunderstandings, I have added the "throw CatalogException"
> > flag
> > >> to the createCatalog method definition of CatalogStore.
> > >>
> > >>> 2. Could you please provide an exhaustive list of the supported
> kinds?
> > >>
> > >> Sure, the documentation now includes the configuration of the
> built-in
> > >> CatalogStore as well as how to configure a custom CatalogStore.
> > >>
> > >>
> > >> Best,
> > >> Feng
> > >>
> > >>
> > >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
> > wrote:
> > >>
> > >>> Hi Feng,
> > >>>
> > >>> Thank you for providing the proposal. I believe this feature will be
> > >>> highly
> > >>> valuable.
> > >>>
> > >>> I have a couple of inquiries:
> > >>>
> > >>> 1. According to the documentation [1], the createCatalog method does
> > not
> > >>> mention any exceptions being thrown. However, I would like to confirm
> > if
> > >>> it
> > >>> is always true that there will be no failures in all scenarios.
> Please
> > let
> > >>> me know if there is any additional information I may have missed.
> > >>>
> > >>> 2. Regarding the registration process using the
> > `table.catalog-store.kind`
> > >>> configuration, could you please provide an exhaustive list of the
> > >>> supported
> > >>> kinds?
> > >>> It would be great to have a comprehensive understanding of the
> > options
> > >>> available.
> > >>>
> > >>> Bests,
> > >>> Samrat
> > >>>
> > >>> [1]
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >>>
> > >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com>
> wrote:
> > >>>
> > >>>> Hi Hang and Jark
> > >>>>
> > >>>> Thank you very much for your reply.
> > >>>>
> > >>>> @Jark
> > >>>>> 1. Could you move the CatalogStore registration API to the "Public
> > >>>> Interface" section?
> > >>>>
> > >>>> Indeed, it is more reasonable.
> > >>>>
> > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > >>>>
> > >>>> Sure, the names that you provided are indeed more logical and
> easier
> > >>> to
> > >>>> understand.
> > >>>>
> > >>>>> 3. About the open/close method in CatalogStoreFactory.
> > >>>>
> > >>>> I also agree with the proposed requirement scenario and design. I
> have
> > >>>> already made modifications to the interface.
> > >>>>
> > >>>> @Hang
> > >>>>
> > >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > close
> > >>>> the resource.
> > >>>>
> > >>>> I have already added the missing method.
> > >>>>
> > >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > >>>>
> > >>>> Sorry for the misspelling, I updated this part.
> > >>>>
> > >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> > methods
> > >>>> to init or close its resource
> > >>>>
> > >>>> This is indeed a scenario that needs to be considered, and I have
> > added
> > >>>> relevant methods accordingly.
> > >>>> Additionally, I have included some relevant description information
> in
> > >>> the
> > >>>> documentation to help others understand
> > >>>> why the open/close methods need to be added.
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>> Feng
> > >>>>
> > >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi Jing,
> > >>>>>
> > >>>>> Thank you for the update.
> > >>>>>
> > >>>>> 1. Could you move the CatalogStore registration API to the "Public
> > >>>>> Interface" section?
> > >>>>> "Proposed Changes" is more like a place to describe the
> > implementation
> > >>>>> details.
> > >>>>>
> > >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> > >>> Besides,
> > >>>>> the config key
> > >>>>> name should be hierarchical[1]. Therefore, it may be better to use:
> > >>>>> "table.catalog-store.kind"
> > >>>>> "table.catalog-store.file.path"
> > >>>>>
> > >>>>> 3. I think Hang's suggestions make sense.
> > >>>>>
> > >>>>> Others look good to me.
> > >>>>>
> > >>>>> Best,
> > >>>>> Jark
> > >>>>>
> > >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> Hi, Feng.
> > >>>>>>
> > >>>>>> Thanks for the update.
> > >>>>>> The current design looks good to me. I have some minor comments.
> > >>>>>>
> > >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> > >>> close
> > >>>>> the
> > >>>>>> resource. For example, when storing the information in MySQL, the
> > >>> store
> > >>>>>> needs to open and close the connections.
> > >>>>>>
> > >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > >>>>>>
> > >>>>>> 3. About the usage in the sql gateway.
> > >>>>>> Considering the usage in sql gateway, the sql gateway may create a
> > >>>>>> CatalogStore for each session.
> > >>>>>> If we are using the MySqlCatalogStore, there would be so many
> > >>>>> connections.
> > >>>>>> How can we reuse the connection among these sessions?
> > >>>>>> I think sql gateway need to maintain a connection pool in
> > >>>>>> the CatalogStoreFactory and each session get its own connection
> from
> > >>>> the
> > >>>>>> pool when it is created.
> > >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
> > >>> to
> > >>>>> init
> > >>>>>> or close its resource.
> > >>>>>> Or is there a better way?
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Hang
> > >>>>>>
> > >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > >>>>>>
> > >>>>>>> Thanks Jingsong.
> > >>>>>>>
> > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> > >>>>>>>
> > >>>>>>> +1 For this, I have already updated FLIP.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Feng
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> > >>> jingsonglee0@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Thanks Feng,
> > >>>>>>>>
> > >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> > >>>>>>>> functions are converted to createxxx from registerxxx.
> > >>>>>>>>
> > >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
> > >>>>
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Hi jark, thanks for your suggestion.
> > >>>>>>>>>
> > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > >>> the
> > >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> > >>>>> Otherwise,
> > >>>>>>>> there
> > >>>>>>>>> might be some data inconsistencies when CatalogStore is
> > >>> changed.
> > >>>>>>>>>
> > >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> > >>>>>> TableEnv.
> > >>>>>>>>> Therefore, my current proposal is to add a method to configure
> > >>>> the
> > >>>>>>>>> CatalogStore in EnvironmentSettings.
> > >>>>>>>>>
> > >>>>>>>>> // Initialize a catalog Store
> > >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> > >>>>>>>>>
> > >>>>>>>>> // set up the Table API
> > >>>>>>>>> final EnvironmentSettings settings =
> > >>>>>>>>>
> > >>> EnvironmentSettings.newInstance().inBatchMode()
> > >>>>>>>>> .withCatalogStore(catalogStore)
> > >>>>>>>>> .build();
> > >>>>>>>>>
> > >>>>>>>>> final TableEnvironment tableEnv =
> > >>>>>>>> TableEnvironment.create(settings);
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > >>>> default
> > >>>>>>>> method,
> > >>>>>>>>> not an interface method?
> > >>>>>>>>>
> > >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> > >>> that
> > >>>> as
> > >>>>> a
> > >>>>>>> new
> > >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> > >>>> method
> > >>>>>> but
> > >>>>>>>> an
> > >>>>>>>>> interface method. I have already modified the interface.
> > >>>>>>>>>
> > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > >>>>>>>>> deprecated`registerCatalog`.
> > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > >>> mention
> > >>>> the
> > >>>>>>>> changed
> > >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > >>>> other
> > >>>>>>>>> persisted catalog store) is used.
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> [1].
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Feng
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Feng,
> > >>>>>>>>>>
> > >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> > >>>>>>>>>> The current design looks pretty good to me. I just have some
> > >>>>> minor
> > >>>>>>>>>> comments.
> > >>>>>>>>>>
> > >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> > >>> the
> > >>>>>>>> CatalogStore
> > >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> > >>> there
> > >>>>>> might
> > >>>>>>> be
> > >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> > >>>>>>>>>>
> > >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> > >>>> default
> > >>>>>>>> method,
> > >>>>>>>>>> not an interface method?
> > >>>>>>>>>>
> > >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> > >>>>> deprecated
> > >>>>>>>>>> `registerCatalog`.
> > >>>>>>>>>>
> > >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> > >>> mention
> > >>>> the
> > >>>>>>>> changed
> > >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> > >>>>> other
> > >>>>>>>>>> persisted catalog store) is used.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Jark
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> > >>> jinfeng1995@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> > >>>>>>>>>>>
> > >>>>>>>>>>> @Ron
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> > >>>> method,
> > >>>>>> do
> > >>>>>>> we
> > >>>>>>>>>> need
> > >>>>>>>>>>> to provide a default implementation?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes, we will provide a default
> > >>> InMemoryCatalogStoreFactory to
> > >>>>>>> create
> > >>>>>>>> an
> > >>>>>>>>>>> InMemoryCatalogStore.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> > >>> initializing
> > >>>>> it,
> > >>>>>>>> whether
> > >>>>>>>>>> we
> > >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> > >>> snapshots,
> > >>>>> and
> > >>>>>>>> once
> > >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> > >>>>>> Catalog>
> > >>>>>>>>>>> catalogs.
> > >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> > >>> the
> > >>>>>> primary
> > >>>>>>>>>> source
> > >>>>>>>>>>> for obtaining the corresponding Catalog.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> > >>>>>>>>>>> `catalog.store.path`?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> > >>> seem a
> > >>>>> bit
> > >>>>>>>>>> strange. I
> > >>>>>>>>>>> will update the FLIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> @Shammon
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thank you for the review. I have made the necessary
> > >>>>> corrections.
> > >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> > >>> I
> > >>>>> have
> > >>>>>>> also
> > >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Feng
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> > >>>> zjureel@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi feng,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for updating, I have some minor comments
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> > >>>>>> `Public
> > >>>>>>>>>>>> Interfaces`, it is not a public interface.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> > >>> and
> > >>>>>>>>>>>> `CatalogStoreFactory`
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> > >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> > >>>>> should
> > >>>>>> be
> > >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> > >>>>>>>>>>>> catalogStore.get(catalogName);`
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> > >>>> ron9.liu@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Feng
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> > >>>> useful
> > >>>>>> for
> > >>>>>>>>>> catalog
> > >>>>>>>>>>>>> management.
> > >>>>>>>>>>>>> I have some small questions:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 1. Regarding the
> > >>> CatalogStoreFactory#createCatalogStore
> > >>>>>> method,
> > >>>>>>>> do we
> > >>>>>>>>>>>> need
> > >>>>>>>>>>>>> to provide a default implementation?
> > >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> > >>>> initializing
> > >>>>>> it,
> > >>>>>>>>>> whether
> > >>>>>>>>>>> we
> > >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> > >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> > >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> > >>> them to
> > >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Ron
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> > >>> 21:19写道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi yuxia
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> > >>>>> register
> > >>>>>>> the
> > >>>>>>>>>>> Catalog,
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>>> initialize it and open it. right?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> > >>>>>> original
> > >>>>>>>>>> CREATE
> > >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> > >>>>>>>>>> registerCatalog
> > >>>>>>>>>>> so
> > >>>>>>>>>>>>>> that parameter validation can be performed.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> > >>>>>>> reflected
> > >>>>>>>> in
> > >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> > >>>>> catalog
> > >>>>>>>>>>>>> configurations,
> > >>>>>>>>>>>>>> only initialization is required in getCatalog.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> > >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
> > >>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi, Feng.
> > >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> > >>>>>>>> initialization*.
> > >>>>>>>>>>> If
> > >>>>>>>>>>>>> i'm
> > >>>>>>>>>>>>>>> wrong, please correct me.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> > >>>> access
> > >>>>>> the
> > >>>>>>>>>>> catalog,
> > >>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> > >>>>> Changes,
> > >>>>>>>> once we
> > >>>>>>>>>>>>>> register
> > >>>>>>>>>>>>>>> the Catalog,
> > >>>>>>>>>>>>>>> we initialize it and open it. right?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>> Yuxia
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ----- 原始邮件 -----
> > >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
> > >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > >>>>>> Catalog
> > >>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>> and asynchronous registration
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> One of the major changes is that current design
> > >>> will
> > >>>>>>> provide
> > >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> > >>>> of a
> > >>>>>>>> cache,
> > >>>>>>>>>>> which
> > >>>>>>>>>>>>>> means
> > >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> > >>>> other
> > >>>>>>>> sessions
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> > >>>>>> further
> > >>>>>>>>>>>> improvement.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > >>>>>>>> jinfeng1995@gmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> > >>>>>> progress
> > >>>>>>>> of
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> > >>> Shammon
> > >>>> FY,
> > >>>>>>>> ShengKai
> > >>>>>>>>>>> Fang
> > >>>>>>>>>>>>>> and I
> > >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> > >>>>>> solution
> > >>>>>>>> for
> > >>>>>>>>>>> Flink
> > >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> > >>>> have
> > >>>>>>>> updated
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> final
> > >>>>>>>>>>>>>>>> solution in FLIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 1.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> > >>>>>>>> configuration
> > >>>>>>>>>>>> similar
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> TableDescriptor.
> > >>>>>>>>>>>>>>>> 2.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> > >>>>>>>>>>> storeCatalog(String
> > >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> > >>>>>> CatalogDescriptor
> > >>>>>>>>>>>>>>> getCatalog(String)
> > >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> > >>>> Catalog
> > >>>>>>>> instance.
> > >>>>>>>>>>>> This
> > >>>>>>>>>>>>>> way,
> > >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> > >>> saving
> > >>>>> and
> > >>>>>>>>>>> retrieving
> > >>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> configurations without having to initialize
> > >>>>>> catalogs.
> > >>>>>>>>>>>>>>>> 3.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The default registerCatalog(String
> > >>> catalogName,
> > >>>>>>> Catalog
> > >>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> > >>>>>>> deprecated.
> > >>>>>>>>>>>>>>>> 4.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> A new function registerCatalog(String
> > >>>> catalogName,
> > >>>>>>>>>>>>> CatalogDescriptor
> > >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> > >>> default
> > >>>>>>>> registration
> > >>>>>>>>>>>>> function
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> > >>>>>>>>>>>>>>>> 5.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> > >>>> will
> > >>>>>>>> remain
> > >>>>>>>>>>>>> unchanged
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> > >>>> deletion
> > >>>>>>>>>> operations
> > >>>>>>>>>>>> from
> > >>>>>>>>>>>>>> one
> > >>>>>>>>>>>>>>>> session won't synchronize with other
> > >>> sessions.
> > >>>>>>>>>>>>>>>> 6.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> To support multi-session synchronization
> > >>>> scenarios
> > >>>>>> for
> > >>>>>>>>>>> deletions
> > >>>>>>>>>>>>>> later
> > >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> > >>>>>>>>>>> configurable.There
> > >>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>> three possible situations:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> > >>>>>>>> automatically
> > >>>>>>>>>>> clear
> > >>>>>>>>>>>>>>>> long-unused catalogs.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> > >>>>>> getCatalog
> > >>>>>>>>>>> creates a
> > >>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>> instance.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is the document for discussion:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This is the final proposal document:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thank you very much for your attention and
> > >>>>> suggestions
> > >>>>>> on
> > >>>>>>>> this
> > >>>>>>>>>>>>> FLIP. A
> > >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> > >>> on
> > >>>>> the
> > >>>>>>>> entire
> > >>>>>>>>>>>> design
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> organizing offline discussions.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> > >>>> about
> > >>>>>>> this
> > >>>>>>>> FLIP
> > >>>>>>>>>>>>> please
> > >>>>>>>>>>>>>>> feel
> > >>>>>>>>>>>>>>>> free to share.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > >>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> > >>> to
> > >>>> me.
> > >>>>>> We
> > >>>>>>>> could
> > >>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> > >>>> common
> > >>>>>>>> built-in
> > >>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> > >>> every
> > >>>>>>>>>> CatalogSotre
> > >>>>>>>>>>>>>>>>> implementation has to take care of it.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > >>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Jing,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for your reply.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > >>>>> suggest
> > >>>>>>> you
> > >>>>>>>>>>>>> completely
> > >>>>>>>>>>>>>>> walk
> > >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> > >>> the
> > >>>>> same
> > >>>>>>>> time, I
> > >>>>>>>>>>>>> greatly
> > >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> > >>>> thoroughly
> > >>>>>>>> checked
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> entire
> > >>>>>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> > >>> FLIP
> > >>>>>>> design,
> > >>>>>>>>>>>>>>> CatalogManager
> > >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> > >>> being
> > >>>>>>>> configured.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> > >>> was
> > >>>> not
> > >>>>>>>> necessary
> > >>>>>>>>>>>>> because
> > >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> > >>>> catalogs
> > >>>>>> to
> > >>>>>>>> store
> > >>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>> instances.
> > >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> > >>>>>>> Therefore, I
> > >>>>>>>>>>>> modified
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> part
> > >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> > >>> Catalog>
> > >>>>>>> catalogs
> > >>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>> CatalogManager.
> > >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> > >>> serve
> > >>>> as
> > >>>>>> the
> > >>>>>>>>>> default
> > >>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> > >>>>>>>> functionality of
> > >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> > >>>>>>>>>>>>>>>>>> The previous plan that kept
> > >>>>>>>> Map<String,Catalog>catalogs has
> > >>>>>>>>>>>> been
> > >>>>>>>>>>>>>>> moved
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > >>>>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There are still many places contain
> > >>>>> inconsistent
> > >>>>>>>> content,
> > >>>>>>>>>>>> e.g.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> > >>> used.
> > >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> > >>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> > >>> not
> > >>>>> tell
> > >>>>>>>> what the
> > >>>>>>>>>>>>> method
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> doing.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> > >>>>> suggest
> > >>>>>>> you
> > >>>>>>>>>>>> completely
> > >>>>>>>>>>>>>>> walk
> > >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> > >>> issues.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> > >>> mean
> > >>>>> that
> > >>>>>>> you
> > >>>>>>>> will
> > >>>>>>>>>>>> build
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> > >>> is a
> > >>>>> very
> > >>>>>>>>>> different
> > >>>>>>>>>>>>>> design
> > >>>>>>>>>>>>>>>>>> concept
> > >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> > >>> I am
> > >>>>> not
> > >>>>>>>>>> mistaken,
> > >>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> > >>>> work
> > >>>>>>>> without
> > >>>>>>>>>>>> Optional
> > >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> > >>>>> reason
> > >>>>>>>> why I
> > >>>>>>>>>>>>> mentioned
> > >>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> > >>>>> Optional
> > >>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>> correct.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > >>>>>>>>>>>>> jinfeng1995@gmail.com>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi Jing
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> > >>>> FLIP
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> > >>>> review. I
> > >>>>>>> have
> > >>>>>>>>>>> already
> > >>>>>>>>>>>>> made
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> relevant changes.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> > >>> the
> > >>>>>>>>>> multi-instance
> > >>>>>>>>>>>>>>>> scenarios, I
> > >>>>>>>>>>>>>>>>>> am
> > >>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> > >>> multiple
> > >>>>>>>> scenarios
> > >>>>>>>>>> with
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> current
> > >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> > >>>>> remove
> > >>>>>>>>>>> 'Map<String,
> > >>>>>>>>>>>>>>>> Catalog>'
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> > >>>>>>>>>> InMemoryCatalogStore.
> > >>>>>>>>>>>> This
> > >>>>>>>>>>>>>>> way,
> > >>>>>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> > >>>>> achieve
> > >>>>>>>>>>>>> multi-instance
> > >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> > >>>> think?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > >>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> > >>> are
> > >>>>>> still
> > >>>>>>>> some
> > >>>>>>>>>>> NIT
> > >>>>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>>>>>> like:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> > >>>> has
> > >>>>>> been
> > >>>>>>>> used
> > >>>>>>>>>> as
> > >>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> > >>> example.
> > >>>> It
> > >>>>>>>> should be
> > >>>>>>>>>>>> fine
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> use
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> > >>> after
> > >>>> you
> > >>>>>>>> submit
> > >>>>>>>>>> the
> > >>>>>>>>>>>> PR.
> > >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> > >>>> somewhere
> > >>>>> in
> > >>>>>>> the
> > >>>>>>>>>>>> rejected
> > >>>>>>>>>>>>>>>> section
> > >>>>>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> > >>> it
> > >>>>>>>> consistent.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> > >>>>>>>> multi-instance
> > >>>>>>>>>>>>>>> scenarios,
> > >>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>> am
> > >>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> > >>> behaviour.
> > >>>> If
> > >>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> > >>>> once
> > >>>>>> the
> > >>>>>>>>>> related
> > >>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> > >>> another
> > >>>>>>>> instance.
> > >>>>>>>>>> Did
> > >>>>>>>>>>> I
> > >>>>>>>>>>>>> miss
> > >>>>>>>>>>>>>>>>>>> something?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> > >>> Jin
> > >>>> <
> > >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> > >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> @Jing
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> > >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> > >>> name, I
> > >>>>>> have
> > >>>>>>>>>> updated
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> documentation.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> > >>> should
> > >>>> be
> > >>>>>>>>>> implemented
> > >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> > >>>>>> alternative
> > >>>>>>> 1
> > >>>>>>>>>> turns
> > >>>>>>>>>>>> out
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> > >>> this
> > >>>>> part
> > >>>>>>> of
> > >>>>>>>> the
> > >>>>>>>>>>>>> content.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> > >>>> forward,
> > >>>>>> the
> > >>>>>>>>>> option 2
> > >>>>>>>>>>>>>>> solution
> > >>>>>>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> > >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> > >>>>> offline,
> > >>>>>>> we
> > >>>>>>>>>> think
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> solution
> > >>>>>>>>>>>>>>>>>> 2
> > >>>>>>>>>>>>>>>>>>>>> might
> > >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> > >>>>>>>> inconsistency
> > >>>>>>>>>>>> issues.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> > >>>>> descriptions
> > >>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>>>> content.
> > >>>>>>>>>>>>>>>> Would
> > >>>>>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> > >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> > >>>> document
> > >>>>>> and
> > >>>>>>>>>>>> appreciate
> > >>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>> suggestions.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> @Shammon
> > >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> > >>> in
> > >>>>>>>> `Rejected
> > >>>>>>>>>>>>>>> Alternatives`
> > >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> > >>> `Rejected
> > >>>>>>>>>> Alternatives`.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best
> > >>>>>>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> > >>> Shammon
> > >>>>> FY
> > >>>>>> <
> > >>>>>>>>>>>>>>> zjureel@gmail.com>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> > >>>>>> `Proposed
> > >>>>>>>>>>> Changes`,
> > >>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>> put
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> > >>>>>>>> Alternatives`? I
> > >>>>>>>>>>> think
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>> help
> > >>>>>>>>>>>>>>>>>>>> us
> > >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> > >>> Jing
> > >>>> Ge
> > >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> > >>> am
> > >>>>>> still
> > >>>>>>>>>> confused
> > >>>>>>>>>>>>> after
> > >>>>>>>>>>>>>>>>>>> completely
> > >>>>>>>>>>>>>>>>>>>>>>> reading
> > >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> > >>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> > >>>>> registerCatalog()
> > >>>>>>> and
> > >>>>>>>>>>>>>> addCatalog()
> > >>>>>>>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>> no
> > >>>>>>>>>>>>>>>>>>>> big
> > >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> > >>>> One
> > >>>>> of
> > >>>>>>>> them is
> > >>>>>>>>>>>>>>> responsible
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> > >>>>>> persistCatalog()?
> > >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> > >>>> Map<String,
> > >>>>>>>> Catalog>
> > >>>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>> as a
> > >>>>>>>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> > >>> data
> > >>>>>>>> persistence.
> > >>>>>>>>>> I
> > >>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>> suggest
> > >>>>>>>>>>>>>>>>>>>>>> describing
> > >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> > >>>>> clearly
> > >>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>> Some
> > >>>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> > >>>> i.e.
> > >>>>>> data
> > >>>>>>>> in
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> store
> > >>>>>>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> > >>>> instance
> > >>>>>>>> should be
> > >>>>>>>>>>>> found
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> store
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> > >>>>>> initialized
> > >>>>>>>> or it
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>> lazy
> > >>>>>>>>>>>>>>>>>>>>>> initialized)
> > >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> > >>>>>>> consistency
> > >>>>>>>> will
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>>> taken
> > >>>>>>>>>>>>>>>> care
> > >>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>> while
> > >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> > >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> > >>>>>> forward,
> > >>>>>>>> the
> > >>>>>>>>>>>> option
> > >>>>>>>>>>>>> 2
> > >>>>>>>>>>>>>>>>> solution
> > >>>>>>>>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> > >>>>>> because,
> > >>>>>>>> afaiu,
> > >>>>>>>>>>>>> issues
> > >>>>>>>>>>>>>>>>>> mentioned
> > >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> > >>>>> solved
> > >>>>>>>> yet. Do
> > >>>>>>>>>>> you
> > >>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>> want
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> propose
> > >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> > >>>> suggestions
> > >>>>>> for
> > >>>>>>>> both
> > >>>>>>>>>> of
> > >>>>>>>>>>>>> them?
> > >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> > >>>> there
> > >>>>>> are
> > >>>>>>>> some
> > >>>>>>>>>>>>>>> inconsistent
> > >>>>>>>>>>>>>>>>>>>>> descriptions
> > >>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> > >>>> clean
> > >>>>>>> them
> > >>>>>>>> up?
> > >>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Jing
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> > >>> Feng
> > >>>>>> Jin <
> > >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> > >>> and
> > >>>> I
> > >>>>>>>> completely
> > >>>>>>>>>>>> agree
> > >>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>> point
> > >>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>> view.
> > >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> > >>>>>> complicated
> > >>>>>>>> the
> > >>>>>>>>>>> whole
> > >>>>>>>>>>>>>> issue.
> > >>>>>>>>>>>>>>>>> First
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> > >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> > >>>> persistence
> > >>>>>> of
> > >>>>>>>> the
> > >>>>>>>>>>>>> Catalog's
> > >>>>>>>>>>>>>>>>>>>> Configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> > >>>>>> persistence
> > >>>>>>>> for
> > >>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>> Configuration,
> > >>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> > >>> to
> > >>>> the
> > >>>>>>>> Catalog
> > >>>>>>>>>>>>>> interface.
> > >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> > >>> Catalog
> > >>>>>>>> instance
> > >>>>>>>>>> to a
> > >>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>> String>
> > >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> > >>>>>>>> implementation
> > >>>>>>>>>> will
> > >>>>>>>>>>>>> throw
> > >>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>> exception.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> > >>>>>>>>>>>>>>>>>>>>>>>>> /**
> > >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> > >>> the
> > >>>>>>>> properties
> > >>>>>>>>>> of
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>> object.
> > >>>>>>>>>>>>>>>>>>>>>>>>> *
> > >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> > >>> the
> > >>>>>>>> properties
> > >>>>>>>>>> of
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>> object
> > >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> > >>>>>>>> UnsupportedOperationException if
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> implementing
> > >>>>>>>>>>>>>>>>>>>>> class
> > >>>>>>>>>>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>>>>>>>>> not override
> > >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> > >>> implementation
> > >>>> of
> > >>>>>> this
> > >>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> */
> > >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> > >>>>>>>> toProperties()
> > >>>>>>>>>> {
> > >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> > >>>>>>>>>>>> UnsupportedOperationException("Please
> > >>>>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> > >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> > >>>> follows:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> > >>> configured a
> > >>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> > >>>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> > >>> registering a
> > >>>>>>> Catalog
> > >>>>>>>>>>>> instance
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > >>>> catalogName,
> > >>>>>>>> Catalog
> > >>>>>>>>>>>>> catalog).
> > >>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> > >>>> Map<String,
> > >>>>>>>> String>
> > >>>>>>>>>>>>> properties
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> saved
> > >>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> > >>>>>> instances
> > >>>>>>>> do
> > >>>>>>>>>> not
> > >>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> method,
> > >>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> > >>>>>>>> CatalogStore,
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> toConfiguration()
> > >>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> > >>>>>>> consistency
> > >>>>>>>> with
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> original
> > >>>>>>>>>>>>>>>>>>>> process.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> > >>>>>> Catalog>
> > >>>>>>>>>>> catalogs
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> > >>>>>> conflicts
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> > >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> > >>>>> itself
> > >>>>>>> and
> > >>>>>>>>>>> provide
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> dedicated
> > >>>>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> > >>> loading,
> > >>>>> such
> > >>>>>>> as
> > >>>>>>>>>>>>>>>>> ConfigurationCatalog
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> public class
> > >>> ConfigurationCatalog
> > >>>>>>>> implements
> > >>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>> {
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>> ConfigurationCatalog(Map<String,
> > >>>>>>> String>
> > >>>>>>>>>>>>> properties)
> > >>>>>>>>>>>>>> {
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> > >>> the
> > >>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> > >>> 10:03 AM
> > >>>>>> Shammon
> > >>>>>>>> FY <
> > >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> > >>> the
> > >>>>>>>> functionality
> > >>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> > >>>>>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> > >>>> multiple
> > >>>>>>>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > >>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> > >>>>>> implementation
> > >>>>>>> in
> > >>>>>>>>>> Trino
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> same,
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> > >>> only
> > >>>>> has
> > >>>>>>> one
> > >>>>>>>>>>>>>>> `createCatalog`,
> > >>>>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>> save
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> > >>>>>> `CatalogStore`.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> > >>>>>> another
> > >>>>>>>>>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> > >>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> > >>> `Map<String,
> > >>>>>>> Catalog>
> > >>>>>>>>>>>> catalogs`
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> > >>>> mentioned
> > >>>>>>>> before,
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>> `Map<String,
> > >>>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> > >>>>>>>> `CatalogManager`.
> > >>>>>>>>>> How
> > >>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>> saving
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> > >>> catalogs`
> > >>>>> and
> > >>>>>>>>>>>> `CatalogStore
> > >>>>>>>>>>>>>>>>>>> catalogStore`
> > >>>>>>>>>>>>>>>>>>>>>>>> together
> > >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> > >>> added
> > >>>> in
> > >>>>>>>>>>>>> `CatalogManager`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > >>> 5:22 PM
> > >>>>> Feng
> > >>>>>>>> Jin <
> > >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> > >>> I
> > >>>> am
> > >>>>>> very
> > >>>>>>>> sorry
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> > >>>>>>>>>>>>>>>>>>>>>>>>> caused
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> > >>>>> original
> > >>>>>>>>>>> discussion.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > >>>>>> pre-discussion
> > >>>>>>>> [1]
> > >>>>>>>>>> for
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> > >>> such a
> > >>>>>>>> discussion
> > >>>>>>>>>>>> before.
> > >>>>>>>>>>>>>>>>> However,
> > >>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> designing
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> > >>> that
> > >>>>> the
> > >>>>>>>> logic of
> > >>>>>>>>>>>>>>>>> CatalogManager
> > >>>>>>>>>>>>>>>>>>>> itself
> > >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> > >>>> cannot
> > >>>>>>> only
> > >>>>>>>>>>> persist
> > >>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>> instances
> > >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> > >>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>>>> might
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> > >>>> Catalogs,
> > >>>>>>>> because
> > >>>>>>>>>> in
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> end
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>>>>>> need
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> save
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> > >>>>> corresponding
> > >>>>>> to
> > >>>>>>>> the
> > >>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>> instances.
> > >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> > >>>>>>>>>>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> > >>>>>>> CatalogStore
> > >>>>>>>>>>>> interface
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> > >>> this
> > >>>>>> part, I
> > >>>>>>>> also
> > >>>>>>>>>>> took
> > >>>>>>>>>>>>>>>>> inspiration
> > >>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > >>>> handling
> > >>>>>>>> catalog
> > >>>>>>>>>> with
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>> looks a
> > >>>>>>>>>>>>>>>>>>>>>>>>> little
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> > >>>> suggestion. I
> > >>>>>>> will
> > >>>>>>>>>>> provide
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>> detailed
> > >>>>>>>>>>>>>>>>>>>>>> description
> > >>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> > >>> part,
> > >>>>> and
> > >>>>>>> add
> > >>>>>>>> it
> > >>>>>>>>>> to
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> > >>>>>>>> `addCatalog`
> > >>>>>>>>>>> still
> > >>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>> confused
> > >>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> > >>>>>> catalog
> > >>>>>>> to
> > >>>>>>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>>>>>> how
> > >>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>> renaming
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> > >>> very
> > >>>>>>> personal
> > >>>>>>>>>>>> opinion,
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>> decide
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> > >>>>> intuitive
> > >>>>>>> to
> > >>>>>>>> me.
> > >>>>>>>>>> I
> > >>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>> see
> > >>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>> problem
> > >>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>> it.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> > >>>>> statement,
> > >>>>>>>> which
> > >>>>>>>>>>>> method
> > >>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> called?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > >>>>>> `addCatalog`?
> > >>>>>>>> I'm
> > >>>>>>>>>>>>> wondering
> > >>>>>>>>>>>>>>>>> whether
> > >>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>> add a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> > >>>>>> stement.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> > >>>> original
> > >>>>>>>> design
> > >>>>>>>>>> was
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> directly
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> > >>> would
> > >>>>>>> disrupt
> > >>>>>>>> the
> > >>>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>> logic.
> > >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> > >>>>>>>>>>>>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> > >>>>>>>> configuration to
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> > >>> the
> > >>>>> same
> > >>>>>>> time
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> > >>> neccessary
> > >>>> to
> > >>>>>>>> provide a
> > >>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>> implmentation
> > >>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > >>>>> `CatalogStoreFactory`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> > >>>>>> otherwise
> > >>>>>>> we
> > >>>>>>>>>> would
> > >>>>>>>>>>>>> need
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> introduce
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> > >>>> configuration
> > >>>>>> for
> > >>>>>>>> lazy
> > >>>>>>>>>>>>> loading.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > >>>>>>> registration
> > >>>>>>>> for
> > >>>>>>>>>>>>> catalog.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> > >>>>>>> registerCatalog(String
> > >>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> > >>>>>> interface
> > >>>>>>>>>> because
> > >>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> > >>>>>>>> initialization
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> > >>>>>>>> inaccurate. It
> > >>>>>>>>>>>>> should
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> lazy
> > >>>>>>>>>>>>>>>>>>>>>>> registration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> > >>>>>>>> registration. I
> > >>>>>>>>>>> have
> > >>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> title
> > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > >>>> 4:27 PM
> > >>>>>>>> Shammon
> > >>>>>>>>>> FY <
> > >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> > >>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>> method
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> > >>>>> method
> > >>>>>>>> named
> > >>>>>>>>>>>>>> `addCatalog`
> > >>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> > >>> the
> > >>>>>> memory
> > >>>>>>>>>> catalog
> > >>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>> `cache`,
> > >>>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> > >>>>>> exposed
> > >>>>>>> to
> > >>>>>>>>>>> users.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> > >>>>>> pre-discussion
> > >>>>>>>> [1]
> > >>>>>>>>>> for
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>> Please
> > >>>>>>>>>>>>>>>>>>>>>>> correct
> > >>>>>>>>>>>>>>>>>>>>>>>>> me
> > >>>>>>>>>>>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> > >>>>> conclusion
> > >>>>>>> of
> > >>>>>>>> that
> > >>>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> use
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> > >>>>>> interface
> > >>>>>>>> and
> > >>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> different
> > >>>>>>>>>>>>>>>>>>>>>>>>> stores
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> > >>>> external
> > >>>>>>>> system.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> > >>>>> between
> > >>>>>>> the
> > >>>>>>>>>>> current
> > >>>>>>>>>>>>> FLIP
> > >>>>>>>>>>>>>>>>> design
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> > >>> of
> > >>>>> the
> > >>>>>>>>>> discussion
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> thread
> > >>>>>>>>>>>>>>>>>> [1] ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> > >>>>> 3:41 PM
> > >>>>>>>> yuxia <
> > >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> > >>> driving
> > >>>>> this
> > >>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> > >>>>> handling
> > >>>>>>>> catalog
> > >>>>>>>>>>> with
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>>> looks a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> little
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> > >>>>> it'll
> > >>>>>> be
> > >>>>>>>>>> better
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> java
> > >>>>>>>>>>>>>>>>>>>>>>>>> doc
> > >>>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> > >>> give a
> > >>>>>> brief
> > >>>>>>>>>> example
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> FLIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> > >>> name
> > >>>>>>>> `addCatalog`
> > >>>>>>>>>>>> still
> > >>>>>>>>>>>>>>> looks
> > >>>>>>>>>>>>>>>>>>> confused
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> > >>>> storing
> > >>>>>>>> catalog to
> > >>>>>>>>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>>>>>>>> how
> > >>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> > >>> It's
> > >>>>> very
> > >>>>>>>> personal
> > >>>>>>>>>>>>>> opinion,
> > >>>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>> decide
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> > >>>>>>> statement,
> > >>>>>>>>>> which
> > >>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> called?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> > >>>>>>>> `addCatalog`?
> > >>>>>>>>>> I'm
> > >>>>>>>>>>>>>>> wondering
> > >>>>>>>>>>>>>>>>>>> whether
> > >>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> > >>> SQL
> > >>>>>>>> stement.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> > >>>> neccessary
> > >>>>> to
> > >>>>>>>>>> provide a
> > >>>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>>> implmentation
> > >>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> > >>>>>>> `CatalogStoreFactory`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> > >>>>>>>> registration
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>> catalog.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> > >>> catalog
> > >>>>>> with
> > >>>>>>>> CREATE
> > >>>>>>>>>>>>>> CATALOG,
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > >>>> used
> > >>>>>> by
> > >>>>>>>>>> default.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> > >>>>> registration
> > >>>>>> is
> > >>>>>>>> the
> > >>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>> behavior,
> > >>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>>>> way
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> > >>> to
> > >>>>>>>> synchronous
> > >>>>>>>>>>>>>>> registration
> > >>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>>>>> before?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> > >>>>> `addCatalog`
> > >>>>>>> and
> > >>>>>>>>>>>>>>>> `registerCatalog`
> > >>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> > >>>>>>>> registration,
> > >>>>>>>>>> it
> > >>>>>>>>>>>> may
> > >>>>>>>>>>>>>> well
> > >>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>> CREATE
> > >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> > >>>> but
> > >>>>>> then
> > >>>>>>>> the
> > >>>>>>>>>>>>> following
> > >>>>>>>>>>>>>>>>> CREATE
> > >>>>>>>>>>>>>>>>>>>> TABLE
> > >>>>>>>>>>>>>>>>>>>>>>>>> statement
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> > >>> fail
> > >>>>> to
> > >>>>>>>> open.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> > >>>> change
> > >>>>>>> which
> > >>>>>>>>>>> should
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>> highlighted
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> > >>> part.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> > >>>> asynchronous
> > >>>>>>>>>>>> registration, I
> > >>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> expect
> > >>>>>>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> > >>>> open
> > >>>>>> or
> > >>>>>>>>>> register
> > >>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> background,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> > >>> comments,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> > >>>> will
> > >>>>> be
> > >>>>>>>>>>> initialized
> > >>>>>>>>>>>>> if
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> > >>>> initialization
> > >>>>>>>> process
> > >>>>>>>>>>> fails,
> > >>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>>>> statements
> > >>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> > >>> successfully."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> > >>> lazy
> > >>>>>>>>>> initialization
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> > >>>>>>>>>> jinfeng1995@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> > >>>>>>> dev@flink.apache.org
> > >>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> > >>> 03日
> > >>>> 下午
> > >>>>>>>> 3:27:45
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> > >>>> 295:
> > >>>>>>>> Support
> > >>>>>>>>>>>>>> persistence
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> > >>>>> registration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> > >>>> all
> > >>>>>> for
> > >>>>>>>> your
> > >>>>>>>>>>>>> interest
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> > >>> catalog
> > >>>>>> with
> > >>>>>>>> the
> > >>>>>>>>>> same
> > >>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> > >>>> crucial
> > >>>>>>>> point.
> > >>>>>>>>>>> Based
> > >>>>>>>>>>>> on
> > >>>>>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>> current
> > >>>>>>>>>>>>>>>>>>>>>> personal
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> > >>>>>> Map<String,
> > >>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>> serve
> > >>>>>>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> > >>> catalogs
> > >>>>> and
> > >>>>>>>> have
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> highest
> > >>>>>>>>>>>>>>>>>>> priority.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> > >>> methods
> > >>>>> that
> > >>>>>>> can
> > >>>>>>>>>> affect
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> > >>> registerCatalog(String
> > >>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> > >>>>> catalog
> > >>>>>>>> instance
> > >>>>>>>>>>>> into
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > >>>> unregisterCatalog(String
> > >>>>>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > >>> the
> > >>>>>> catalog
> > >>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>> corresponding
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> > >>>>> Catalog>
> > >>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> > >>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> > >>>> retrieves
> > >>>>>> the
> > >>>>>>>>>>>>> corresponding
> > >>>>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> > >>> Catalog>
> > >>>>>>>> catalogs. If
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>> exist,
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> > >>>> corresponding
> > >>>>>>>>>>> configuration
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> > >>> puts
> > >>>>> the
> > >>>>>>>>>>> initialized
> > >>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > >>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> > >>> methods
> > >>>>>> only
> > >>>>>>>> modify
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> > >>>>>>> catalogName,
> > >>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>> String>
> > >>>>>>>>>>>>>>>>>>>>>>> properties)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> > >>>>>>> properties
> > >>>>>>>> to
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> catalogStore
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> checks
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> > >>>>>> catalogName
> > >>>>>>>> with
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>> name.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> > >>> removeCatalog(String
> > >>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> > >>> the
> > >>>>>>> specified
> > >>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> specified
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> > >>>>>>> catalogStore.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> > >>>> possible
> > >>>>>>>> conflict
> > >>>>>>>>>>>>>> scenarios:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> > >>> corresponding
> > >>>>>>>> catalogName
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> > >>> in
> > >>>> the
> > >>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>> Catalog>,
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > >>>>>>>> catalogName,
> > >>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> > >>> directly
> > >>>>> saved
> > >>>>>>> to
> > >>>>>>>> the
> > >>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> > >>> corresponding
> > >>>>>>>> catalogName
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> > >>>>>>> Map<String,
> > >>>>>>>>>>>> Catalog>,
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> > >>>>> catalog)
> > >>>>>>>> method
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>>> fail.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> > >>> corresponding
> > >>>>>>>> catalogName
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> exists
> > >>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> > >>> the
> > >>>>>>>>>>>> addCatalog(String
> > >>>>>>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> > >>>> method
> > >>>>>> can
> > >>>>>>>>>> directly
> > >>>>>>>>>>>>> save
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> properties
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> > >>>>>>>>>> getCatalog(String
> > >>>>>>>>>>>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> > >>>> for
> > >>>>>>>>>>> initialization
> > >>>>>>>>>>>>>>> because
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> > >>> already
> > >>>>>> exists
> > >>>>>>>> in
> > >>>>>>>>>>>> catalogs
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> > >>>>>>>>>>>>> unregisterCatalog(String
> > >>>>>>>>>>>>>>>>>>>> catalogName)
> > >>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> > >>>>>>>> corresponding to
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> original
> > >>>>>>>>>>>>>>>>>>>>> catalogName
> > >>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> > >>> confuse
> > >>>>>> users
> > >>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> > >>>>> catalog)`
> > >>>>>>> in
> > >>>>>>>> the
> > >>>>>>>>>>>>>>> `Map<String,
> > >>>>>>>>>>>>>>>>>>>> Catalog>
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>> `registerCatalog(String
> > >>>>>>>> catalogName,
> > >>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>> String>
> > >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> > >>>> lead
> > >>>>>> to
> > >>>>>>>>>>>> confusion. I
> > >>>>>>>>>>>>>>>> suggest
> > >>>>>>>>>>>>>>>>>>>>> changing
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> > >>> to
> > >>>>>>>>>>> addCatalog(String
> > >>>>>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> > >>> as
> > >>>>>>>> previously
> > >>>>>>>>>>>>> mentioned
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> > >>>>> `registerCatalog(String
> > >>>>>>>>>>>>>>> catalogName,Catalog
> > >>>>>>>>>>>>>>>>>>>> catalog,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> > >>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> > >>>> already
> > >>>>> an
> > >>>>>>>>>> instance,
> > >>>>>>>>>>>>>> adding
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> > >>>>>>>>>> registerCatalog(String
> > >>>>>>>>>>>>>>>>> catalogName,
> > >>>>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> > >>>> necessarily
> > >>>>>>>> result in
> > >>>>>>>>>>> lazy
> > >>>>>>>>>>>>>>>>>>> initialization.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> > >>>> about
> > >>>>>>>>>> encryption
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> > >>>>>>> necessary,
> > >>>>>>>> but
> > >>>>>>>>>>>>> perhaps
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> encryption
> > >>>>>>>>>>>>>>>>>>>>>>> logic
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> > >>> in
> > >>>>> the
> > >>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>> actually
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> > >>> the
> > >>>>>>>>>>> FileCatalogStore
> > >>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>> encrypt
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> > >>>>> while
> > >>>>>>> the
> > >>>>>>>>>>>>>>>> MemoryCatalogStore
> > >>>>>>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>> require
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> > >>> the
> > >>>>>>>>>>>>>> `MemoryCatalogStore`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> > >>> necessary
> > >>>> to
> > >>>>>>> have a
> > >>>>>>>>>>>>>>>>> MemoryCatalogStore
> > >>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> default
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> > >>>> saves
> > >>>>>>>> Catalog
> > >>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> memory.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> > >>> to
> > >>>>>>>> implement
> > >>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>> loading
> > >>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> > >>> introduce
> > >>>>>>>> additional
> > >>>>>>>>>>> cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> > >>>>>>>> registration
> > >>>>>>>>>>>> failed?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> > >>> critical
> > >>>>>>> concern.
> > >>>>>>>>>> When
> > >>>>>>>>>>>>>>> executing
> > >>>>>>>>>>>>>>>>> DDL,
> > >>>>>>>>>>>>>>>>>>>> DQL,
> > >>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>> DML
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> > >>>> reference a
> > >>>>>>>> specific
> > >>>>>>>>>>>>> Catalog,
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> Catalog
> > >>>>>>>>>>>>>>>>>>>>>>>> instance
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> > >>> if it
> > >>>>> has
> > >>>>>>> not
> > >>>>>>>>>> been
> > >>>>>>>>>>>>>>>> initialized
> > >>>>>>>>>>>>>>>>>> yet.
> > >>>>>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> > >>>>> fails,
> > >>>>>>>> these
> > >>>>>>>>>>>>>> statements
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> executed
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> > >>>>>> Catalog>
> > >>>>>>>>>> catalogs
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> > >>>> have
> > >>>>>> the
> > >>>>>>>> same
> > >>>>>>>>>>>>> catalog
> > >>>>>>>>>>>>>>>> name,
> > >>>>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>>>> correponding
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > >>>>> instances.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> > >>>> resolved
> > >>>>> by
> > >>>>>>>>>> referring
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> previous
> > >>>>>>>>>>>>>>>>>>>>>>>> responses.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> > >>> the
> > >>>>>>> Catalog
> > >>>>>>>>>> that
> > >>>>>>>>>>>> has
> > >>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>> recently
> > >>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> > >>>> for
> > >>>>>>>> subsequent
> > >>>>>>>>>>>> use.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> > >>>>>> 10:58 PM
> > >>>>>>>>>> Xianxun
> > >>>>>>>>>>>> Ye
> > >>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> > >>> this
> > >>>>>> Flip,
> > >>>>>>>> I do
> > >>>>>>>>>>>>> believe
> > >>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> Flip
> > >>>>>>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> > >>>> work
> > >>>>>> for
> > >>>>>>>> also
> > >>>>>>>>>>>>> manages a
> > >>>>>>>>>>>>>>> lot
> > >>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>> catalogs,
> > >>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> > >>>> becomes
> > >>>>>>> slow
> > >>>>>>>>>>> because
> > >>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>> loading a
> > >>>>>>>>>>>>>>>>>>>>> number
> > >>>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> > >>> catalogs in
> > >>>>> the
> > >>>>>>>> user's
> > >>>>>>>>>>>> Flink
> > >>>>>>>>>>>>>> SQL
> > >>>>>>>>>>>>>>>>>> through
> > >>>>>>>>>>>>>>>>>>>>>> regular
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> > >>>>> catalogs
> > >>>>>>> to
> > >>>>>>>>>>> improve
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> speed
> > >>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>> task
> > >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> > >>> flip-295, I
> > >>>>>> have
> > >>>>>>>> some
> > >>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> > >>>>> catalog
> > >>>>>>> with
> > >>>>>>>>>>> CREATE
> > >>>>>>>>>>>>>>> CATALOG,
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> > >>>> used
> > >>>>>> by
> > >>>>>>>>>> default.
> > >>>>>>>>>>>>> What
> > >>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> > >>>>> implement
> > >>>>>>>>>>> asynchronous
> > >>>>>>>>>>>>>>>>>> registration?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> > >>> same
> > >>>>>>>> question
> > >>>>>>>>>> with
> > >>>>>>>>>>>>>>> Shammon’s
> > >>>>>>>>>>>>>>>>>> first
> > >>>>>>>>>>>>>>>>>>>>>>> comment.
> > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> > >>>>> catalogs
> > >>>>>>> and
> > >>>>>>>>>>>>>> CatalogStore
> > >>>>>>>>>>>>>>>>>>>> catalogstore
> > >>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> > >>> have
> > >>>> the
> > >>>>>>> same
> > >>>>>>>>>>> catalog
> > >>>>>>>>>>>>>> name,
> > >>>>>>>>>>>>>>>> but
> > >>>>>>>>>>>>>>>>>>>>>> correponding
> > >>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> > >>>>> instances.
> > >>>>>>> For
> > >>>>>>>>>>> example:
> > >>>>>>>>>>>>>>>>> catalog1->
> > >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> > >>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> > >>> catalog1 ->
> > >>>>>>>> hiveCatalog
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> catalogstore.
> > >>>>>>>>>>>>>>>>>>>>>>> Would
> > >>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> > >>> should
> > >>>>>> define
> > >>>>>>>>>>> `catalogs`
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> > >>>>>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> > >>> 11:13，Hang
> > >>>>>> Ruan
> > >>>>>>> <
> > >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 写道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Leonard,
Thanks for your reply.
> 1. a How to construct a CatalogDescriptor ?
I think it would be helpful to add a method for constructing a
CatalogDescriptor, as you mentioned in 1.c. I will update the documentation
later.
> 1.b How to visit the fields ? Could we use Configuration instead of
Map<String, String> ?
I believe that the use of Map<String, String> options is only intended for
creating a catalog and not for accessing internal parameters.
Since all of the relevant parameters for CREATE CATALOG are also stored in
Map<String, String> options, my understanding is that using Map<String,
String> options should suffice.
Here is the implementation of execute CREATE CATALOG statement.
```java
private TableResultInternal createCatalog(CreateCatalogOperation operation)
{
String exMsg = getDDLOpExecuteErrorMsg(operation.asSummaryString());
try {
String catalogName = operation.getCatalogName();
Map<String, String> properties = operation.getProperties();
Catalog catalog =
FactoryUtil.createCatalog(
catalogName,
properties,
tableConfig,
resourceManager.getUserClassLoader());
catalogManager.registerCatalog(catalogName, catalog);
return TableResultImpl.TABLE_RESULT_OK;
} catch (CatalogException e) {
throw new ValidationException(exMsg, e);
}
}
```
> 2. Do we have plan to offer a default CatalogStore if user didn’t config
this?
Yes, the in-memory catalogStore will be used as the default CatalogStore
even if the user has not configured one
Best,
Feng
On Tue, Jun 6, 2023 at 8:02 PM Leonard Xu <xb...@gmail.com> wrote:
> Hi, Feng
>
> Sorry for reply late, but I’ve some comments about the FLIP
>
>
> 1. The introduced Public class CatalogDescriptor seems missed some
> necessary component
> a) How to construct a CatalogDescriptor ?
> b) How to visit the fields ? Could we use Configuration instead of
> Map<String, String> ?
> c) Could we offer a built-in factory method to build a CatalogDescriptor
> like
> public static CatalogDescriptor of(String catalogName, Configuration
> configuration)
>
> 2. The FLIP said “By default, there are two built-in CatalogStores
> available: the In-Memory CatalogStore and the File CatalogStore” ,
> Do we have plan to offer a default CatalogStore if user didn’t config
> this? IIUC, users can obtains the benefits from lazy catalog
> initialization If
> we have a default catalogstore even it is in-memory.
>
> Best,
> Leonard
>
>
>
> > On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > Thank you everyone for your valuable input. If there are no further
> > questions or concerns about the FLIP[1], I would like to start voting
> > tomorrow (6/7).
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >
> >
> > Best,
> > Feng
> >
> >
> > On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
> >
> >> Hi Samrat,
> >>
> >> Thanks for your advice.
> >>
> >>> 1. The createCatalog method does not mention any exceptions being
> >> thrown.
> >>
> >> CreateCatalog will throw CatalogException like registerCatalog. As
> >> CatalogException is a RuntimeException,
> >> there is no explicit declaration of throwing Exceptions in
> CatalogManager
> >> and TableEnvironment.
> >> To avoid misunderstandings, I have added the "throw CatalogException"
> flag
> >> to the createCatalog method definition of CatalogStore.
> >>
> >>> 2. Could you please provide an exhaustive list of the supported kinds?
> >>
> >> Sure, the documentation now includes the configuration of the built-in
> >> CatalogStore as well as how to configure a custom CatalogStore.
> >>
> >>
> >> Best,
> >> Feng
> >>
> >>
> >> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com>
> wrote:
> >>
> >>> Hi Feng,
> >>>
> >>> Thank you for providing the proposal. I believe this feature will be
> >>> highly
> >>> valuable.
> >>>
> >>> I have a couple of inquiries:
> >>>
> >>> 1. According to the documentation [1], the createCatalog method does
> not
> >>> mention any exceptions being thrown. However, I would like to confirm
> if
> >>> it
> >>> is always true that there will be no failures in all scenarios. Please
> let
> >>> me know if there is any additional information I may have missed.
> >>>
> >>> 2. Regarding the registration process using the
> `table.catalog-store.kind`
> >>> configuration, could you please provide an exhaustive list of the
> >>> supported
> >>> kinds?
> >>> It would be great to have a comprehensive understanding of the
> options
> >>> available.
> >>>
> >>> Bests,
> >>> Samrat
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>
> >>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
> >>>
> >>>> Hi Hang and Jark
> >>>>
> >>>> Thank you very much for your reply.
> >>>>
> >>>> @Jark
> >>>>> 1. Could you move the CatalogStore registration API to the "Public
> >>>> Interface" section?
> >>>>
> >>>> Indeed, it is more reasonable.
> >>>>
> >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> >>>>
> >>>> Sure, the names that you provided are indeed more logical and easier
> >>> to
> >>>> understand.
> >>>>
> >>>>> 3. About the open/close method in CatalogStoreFactory.
> >>>>
> >>>> I also agree with the proposed requirement scenario and design. I have
> >>>> already made modifications to the interface.
> >>>>
> >>>> @Hang
> >>>>
> >>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> close
> >>>> the resource.
> >>>>
> >>>> I have already added the missing method.
> >>>>
> >>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >>>>
> >>>> Sorry for the misspelling, I updated this part.
> >>>>
> >>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close`
> methods
> >>>> to init or close its resource
> >>>>
> >>>> This is indeed a scenario that needs to be considered, and I have
> added
> >>>> relevant methods accordingly.
> >>>> Additionally, I have included some relevant description information in
> >>> the
> >>>> documentation to help others understand
> >>>> why the open/close methods need to be added.
> >>>>
> >>>>
> >>>> Best,
> >>>> Feng
> >>>>
> >>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> >>>>
> >>>>> Hi Jing,
> >>>>>
> >>>>> Thank you for the update.
> >>>>>
> >>>>> 1. Could you move the CatalogStore registration API to the "Public
> >>>>> Interface" section?
> >>>>> "Proposed Changes" is more like a place to describe the
> implementation
> >>>>> details.
> >>>>>
> >>>>> 2. We should prefix "table." for the CatalogStore configuration.
> >>> Besides,
> >>>>> the config key
> >>>>> name should be hierarchical[1]. Therefore, it may be better to use:
> >>>>> "table.catalog-store.kind"
> >>>>> "table.catalog-store.file.path"
> >>>>>
> >>>>> 3. I think Hang's suggestions make sense.
> >>>>>
> >>>>> Others look good to me.
> >>>>>
> >>>>> Best,
> >>>>> Jark
> >>>>>
> >>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> Hi, Feng.
> >>>>>>
> >>>>>> Thanks for the update.
> >>>>>> The current design looks good to me. I have some minor comments.
> >>>>>>
> >>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
> >>> close
> >>>>> the
> >>>>>> resource. For example, when storing the information in MySQL, the
> >>> store
> >>>>>> needs to open and close the connections.
> >>>>>>
> >>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >>>>>>
> >>>>>> 3. About the usage in the sql gateway.
> >>>>>> Considering the usage in sql gateway, the sql gateway may create a
> >>>>>> CatalogStore for each session.
> >>>>>> If we are using the MySqlCatalogStore, there would be so many
> >>>>> connections.
> >>>>>> How can we reuse the connection among these sessions?
> >>>>>> I think sql gateway need to maintain a connection pool in
> >>>>>> the CatalogStoreFactory and each session get its own connection from
> >>>> the
> >>>>>> pool when it is created.
> >>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
> >>> to
> >>>>> init
> >>>>>> or close its resource.
> >>>>>> Or is there a better way?
> >>>>>>
> >>>>>> Best,
> >>>>>> Hang
> >>>>>>
> >>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> >>>>>>
> >>>>>>> Thanks Jingsong.
> >>>>>>>
> >>>>>>>> Just naming, maybe `createCatalog` in TableEnv
> >>>>>>>
> >>>>>>> +1 For this, I have already updated FLIP.
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Feng
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> >>> jingsonglee0@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Feng,
> >>>>>>>>
> >>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
> >>>>>>>> functions are converted to createxxx from registerxxx.
> >>>>>>>>
> >>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi jark, thanks for your suggestion.
> >>>>>>>>>
> >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> >>> the
> >>>>>>>>> CatalogStore should be immutable once TableEnv is created.
> >>>>> Otherwise,
> >>>>>>>> there
> >>>>>>>>> might be some data inconsistencies when CatalogStore is
> >>> changed.
> >>>>>>>>>
> >>>>>>>>> Yes, We should initialize the CatalogStore when creating the
> >>>>>> TableEnv.
> >>>>>>>>> Therefore, my current proposal is to add a method to configure
> >>>> the
> >>>>>>>>> CatalogStore in EnvironmentSettings.
> >>>>>>>>>
> >>>>>>>>> // Initialize a catalog Store
> >>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
> >>>>>>>>>
> >>>>>>>>> // set up the Table API
> >>>>>>>>> final EnvironmentSettings settings =
> >>>>>>>>>
> >>> EnvironmentSettings.newInstance().inBatchMode()
> >>>>>>>>> .withCatalogStore(catalogStore)
> >>>>>>>>> .build();
> >>>>>>>>>
> >>>>>>>>> final TableEnvironment tableEnv =
> >>>>>>>> TableEnvironment.create(settings);
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> >>>> default
> >>>>>>>> method,
> >>>>>>>>> not an interface method?
> >>>>>>>>>
> >>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
> >>> that
> >>>> as
> >>>>> a
> >>>>>>> new
> >>>>>>>>> interface, the CatalogStoreFactory should not have a default
> >>>> method
> >>>>>> but
> >>>>>>>> an
> >>>>>>>>> interface method. I have already modified the interface.
> >>>>>>>>>
> >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> >>>>>>>>> deprecated`registerCatalog`.
> >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> >>> mention
> >>>> the
> >>>>>>>> changed
> >>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> >>>> other
> >>>>>>>>> persisted catalog store) is used.
> >>>>>>>>>
> >>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1].
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Feng
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Feng,
> >>>>>>>>>>
> >>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
> >>>>>>>>>> The current design looks pretty good to me. I just have some
> >>>>> minor
> >>>>>>>>>> comments.
> >>>>>>>>>>
> >>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
> >>> the
> >>>>>>>> CatalogStore
> >>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
> >>> there
> >>>>>> might
> >>>>>>> be
> >>>>>>>>>> some data inconsistencies when CatalogStore is changed.
> >>>>>>>>>>
> >>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
> >>>> default
> >>>>>>>> method,
> >>>>>>>>>> not an interface method?
> >>>>>>>>>>
> >>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
> >>>>> deprecated
> >>>>>>>>>> `registerCatalog`.
> >>>>>>>>>>
> >>>>>>>>>> 4. In the "Compatibility" section, would be better to
> >>> mention
> >>>> the
> >>>>>>>> changed
> >>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
> >>>>> other
> >>>>>>>>>> persisted catalog store) is used.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Jark
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
> >>> jinfeng1995@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi ， thanks all for reviewing the flip.
> >>>>>>>>>>>
> >>>>>>>>>>> @Ron
> >>>>>>>>>>>
> >>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
> >>>> method,
> >>>>>> do
> >>>>>>> we
> >>>>>>>>>> need
> >>>>>>>>>>> to provide a default implementation?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, we will provide a default
> >>> InMemoryCatalogStoreFactory to
> >>>>>>> create
> >>>>>>>> an
> >>>>>>>>>>> InMemoryCatalogStore.
> >>>>>>>>>>>
> >>>>>>>>>>>> If we get a Catalog from CatalogStore, after
> >>> initializing
> >>>>> it,
> >>>>>>>> whether
> >>>>>>>>>> we
> >>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, in the current design, catalogs are stored as
> >>> snapshots,
> >>>>> and
> >>>>>>>> once
> >>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
> >>>>>> Catalog>
> >>>>>>>>>>> catalogs.
> >>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
> >>> the
> >>>>>> primary
> >>>>>>>>>> source
> >>>>>>>>>>> for obtaining the corresponding Catalog.
> >>>>>>>>>>>
> >>>>>>>>>>>> how about renaming them to `catalog.store.type` and
> >>>>>>>>>>> `catalog.store.path`?
> >>>>>>>>>>>
> >>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
> >>> seem a
> >>>>> bit
> >>>>>>>>>> strange. I
> >>>>>>>>>>> will update the FLIP.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> @Shammon
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you for the review. I have made the necessary
> >>>>> corrections.
> >>>>>>>>>>> Regarding the modifications made to the Public Interface,
> >>> I
> >>>>> have
> >>>>>>> also
> >>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Feng
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> >>>> zjureel@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi feng,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for updating, I have some minor comments
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
> >>>>>> `Public
> >>>>>>>>>>>> Interfaces`, it is not a public interface.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
> >>> and
> >>>>>>>>>>>> `CatalogStoreFactory`
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
> >>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
> >>>>> should
> >>>>>> be
> >>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
> >>>>>>>>>>>> catalogStore.get(catalogName);`
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
> >>>> ron9.liu@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Feng
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
> >>>> useful
> >>>>>> for
> >>>>>>>>>> catalog
> >>>>>>>>>>>>> management.
> >>>>>>>>>>>>> I have some small questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1. Regarding the
> >>> CatalogStoreFactory#createCatalogStore
> >>>>>> method,
> >>>>>>>> do we
> >>>>>>>>>>>> need
> >>>>>>>>>>>>> to provide a default implementation?
> >>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
> >>>> initializing
> >>>>>> it,
> >>>>>>>>>> whether
> >>>>>>>>>>> we
> >>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
> >>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
> >>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
> >>> them to
> >>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> >>> 21:19写道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi yuxia
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
> >>>>> register
> >>>>>>> the
> >>>>>>>>>>> Catalog,
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>> initialize it and open it. right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
> >>>>>> original
> >>>>>>>>>> CREATE
> >>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
> >>>>>>>>>> registerCatalog
> >>>>>>>>>>> so
> >>>>>>>>>>>>>> that parameter validation can be performed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In the current design, lazy initialization is mainly
> >>>>>>> reflected
> >>>>>>>> in
> >>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
> >>>>> catalog
> >>>>>>>>>>>>> configurations,
> >>>>>>>>>>>>>> only initialization is required in getCatalog.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
> >>>>>>>> luoyuxia@alumni.sjtu.edu.cn
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Feng.
> >>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
> >>>>>>>> initialization*.
> >>>>>>>>>>> If
> >>>>>>>>>>>>> i'm
> >>>>>>>>>>>>>>> wrong, please correct me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
> >>>> access
> >>>>>> the
> >>>>>>>>>>> catalog,
> >>>>>>>>>>>>> then
> >>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
> >>>>> Changes,
> >>>>>>>> once we
> >>>>>>>>>>>>>> register
> >>>>>>>>>>>>>>> the Catalog,
> >>>>>>>>>>>>>>> we initialize it and open it. right?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Yuxia
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ----- 原始邮件 -----
> >>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> >>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
> >>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> >>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> >>>>>> Catalog
> >>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>> and asynchronous registration
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> One of the major changes is that current design
> >>> will
> >>>>>>> provide
> >>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
> >>>> of a
> >>>>>>>> cache,
> >>>>>>>>>>> which
> >>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>> once it has been initialized, any changes done by
> >>>> other
> >>>>>>>> sessions
> >>>>>>>>>>> will
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
> >>>>>> further
> >>>>>>>>>>>> improvement.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> >>>>>>>> jinfeng1995@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
> >>>>>> progress
> >>>>>>>> of
> >>>>>>>>>> the
> >>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
> >>> Shammon
> >>>> FY,
> >>>>>>>> ShengKai
> >>>>>>>>>>> Fang
> >>>>>>>>>>>>>> and I
> >>>>>>>>>>>>>>>> had an offline discussion regarding the overall
> >>>>>> solution
> >>>>>>>> for
> >>>>>>>>>>> Flink
> >>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
> >>>> have
> >>>>>>>> updated
> >>>>>>>>>> the
> >>>>>>>>>>>>> final
> >>>>>>>>>>>>>>>> solution in FLIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 1.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
> >>>>>>>> configuration
> >>>>>>>>>>>> similar
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> TableDescriptor.
> >>>>>>>>>>>>>>>> 2.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
> >>>>>>>>>>> storeCatalog(String
> >>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
> >>>>>> CatalogDescriptor
> >>>>>>>>>>>>>>> getCatalog(String)
> >>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
> >>>> Catalog
> >>>>>>>> instance.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>> CatalogStore will only be responsible for
> >>> saving
> >>>>> and
> >>>>>>>>>>> retrieving
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> configurations without having to initialize
> >>>>>> catalogs.
> >>>>>>>>>>>>>>>> 3.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> The default registerCatalog(String
> >>> catalogName,
> >>>>>>> Catalog
> >>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>> function in CatalogManager will be marked as
> >>>>>>> deprecated.
> >>>>>>>>>>>>>>>> 4.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> A new function registerCatalog(String
> >>>> catalogName,
> >>>>>>>>>>>>> CatalogDescriptor
> >>>>>>>>>>>>>>>> catalog) will be added to serve as the
> >>> default
> >>>>>>>> registration
> >>>>>>>>>>>>> function
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> catalogs in CatalogManager.
> >>>>>>>>>>>>>>>> 5.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
> >>>> will
> >>>>>>>> remain
> >>>>>>>>>>>>> unchanged
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> save initialized catalogs.This means that
> >>>> deletion
> >>>>>>>>>> operations
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>> session won't synchronize with other
> >>> sessions.
> >>>>>>>>>>>>>>>> 6.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> To support multi-session synchronization
> >>>> scenarios
> >>>>>> for
> >>>>>>>>>>> deletions
> >>>>>>>>>>>>>> later
> >>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
> >>>>>>>>>>> configurable.There
> >>>>>>>>>>>>> may
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> three possible situations:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
> >>>>>>>> automatically
> >>>>>>>>>>> clear
> >>>>>>>>>>>>>>>> long-unused catalogs.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> c.No caching of any instances; each call to
> >>>>>> getCatalog
> >>>>>>>>>>> creates a
> >>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>> instance.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is the document for discussion:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This is the final proposal document:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you very much for your attention and
> >>>>> suggestions
> >>>>>> on
> >>>>>>>> this
> >>>>>>>>>>>>> FLIP. A
> >>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
> >>> on
> >>>>> the
> >>>>>>>> entire
> >>>>>>>>>>>> design
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> organizing offline discussions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If you have any further suggestions or feedback
> >>>> about
> >>>>>>> this
> >>>>>>>> FLIP
> >>>>>>>>>>>>> please
> >>>>>>>>>>>>>>> feel
> >>>>>>>>>>>>>>>> free to share.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
> >>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
> >>> to
> >>>> me.
> >>>>>> We
> >>>>>>>> could
> >>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> reconsider in the future how to provide more
> >>>> common
> >>>>>>>> built-in
> >>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
> >>> every
> >>>>>>>>>> CatalogSotre
> >>>>>>>>>>>>>>>>> implementation has to take care of it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> >>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jing,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> >>>>> suggest
> >>>>>>> you
> >>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>> walk
> >>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
> >>> the
> >>>>> same
> >>>>>>>> time, I
> >>>>>>>>>>>>> greatly
> >>>>>>>>>>>>>>>>>> appreciate your careful review. I have
> >>>> thoroughly
> >>>>>>>> checked
> >>>>>>>>>> the
> >>>>>>>>>>>>>> entire
> >>>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>> and made corrections to these issues.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
> >>> FLIP
> >>>>>>> design,
> >>>>>>>>>>>>>>> CatalogManager
> >>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
> >>> being
> >>>>>>>> configured.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
> >>> was
> >>>> not
> >>>>>>>> necessary
> >>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
> >>>> catalogs
> >>>>>> to
> >>>>>>>> store
> >>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> instances.
> >>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
> >>>>>>> Therefore, I
> >>>>>>>>>>>> modified
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> part
> >>>>>>>>>>>>>>>>>> of the design and removed Map<String,
> >>> Catalog>
> >>>>>>> catalogs
> >>>>>>>>>> from
> >>>>>>>>>>>>>>>>>> CatalogManager.
> >>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
> >>> serve
> >>>> as
> >>>>>> the
> >>>>>>>>>> default
> >>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
> >>>>>>>> functionality of
> >>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
> >>>>>>>>>>>>>>>>>> The previous plan that kept
> >>>>>>>> Map<String,Catalog>catalogs has
> >>>>>>>>>>>> been
> >>>>>>>>>>>>>>> moved
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> Rejected Alternatives.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> >>>>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There are still many places contain
> >>>>> inconsistent
> >>>>>>>> content,
> >>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
> >>> used.
> >>>>>>>>>>>>>>>>>>> 2. The java comment of the method
> >>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
> >>> not
> >>>>> tell
> >>>>>>>> what the
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> doing.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> There might be more such issues. I would
> >>>>> suggest
> >>>>>>> you
> >>>>>>>>>>>> completely
> >>>>>>>>>>>>>>> walk
> >>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
> >>> issues.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
> >>> mean
> >>>>> that
> >>>>>>> you
> >>>>>>>> will
> >>>>>>>>>>>> build
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
> >>> is a
> >>>>> very
> >>>>>>>>>> different
> >>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>> concept
> >>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
> >>> I am
> >>>>> not
> >>>>>>>>>> mistaken,
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
> >>>> work
> >>>>>>>> without
> >>>>>>>>>>>> Optional
> >>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
> >>>>> reason
> >>>>>>>> why I
> >>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> last email that the example code wrt the
> >>>>> Optional
> >>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> correct.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> >>>>>>>>>>>>> jinfeng1995@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Jing
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
> >>>> FLIP
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
> >>>> review. I
> >>>>>>> have
> >>>>>>>>>>> already
> >>>>>>>>>>>>> made
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> relevant changes.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
> >>> the
> >>>>>>>>>> multi-instance
> >>>>>>>>>>>>>>>> scenarios, I
> >>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
> >>> multiple
> >>>>>>>> scenarios
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
> >>>>> remove
> >>>>>>>>>>> 'Map<String,
> >>>>>>>>>>>>>>>> Catalog>'
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
> >>>>>>>>>> InMemoryCatalogStore.
> >>>>>>>>>>>> This
> >>>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
> >>>>> achieve
> >>>>>>>>>>>>> multi-instance
> >>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
> >>>> think?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> >>>>>>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
> >>> are
> >>>>>> still
> >>>>>>>> some
> >>>>>>>>>>> NIT
> >>>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>>>>> like:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
> >>>> has
> >>>>>> been
> >>>>>>>> used
> >>>>>>>>>> as
> >>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
> >>> example.
> >>>> It
> >>>>>>>> should be
> >>>>>>>>>>>> fine
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
> >>> after
> >>>> you
> >>>>>>>> submit
> >>>>>>>>>> the
> >>>>>>>>>>>> PR.
> >>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
> >>>> somewhere
> >>>>> in
> >>>>>>> the
> >>>>>>>>>>>> rejected
> >>>>>>>>>>>>>>>> section
> >>>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
> >>> it
> >>>>>>>> consistent.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
> >>>>>>>> multi-instance
> >>>>>>>>>>>>>>> scenarios,
> >>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> sure if this is the intended
> >>> behaviour.
> >>>> If
> >>>>>>>>>> Map<String,
> >>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
> >>>> once
> >>>>>> the
> >>>>>>>>>> related
> >>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
> >>> another
> >>>>>>>> instance.
> >>>>>>>>>> Did
> >>>>>>>>>>> I
> >>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>> something?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
> >>> Jin
> >>>> <
> >>>>>>>>>>>>>>> jinfeng1995@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
> >>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> @Jing
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
> >>>>>>>>>>>>>>>>>>>>>> I think this is a good function
> >>> name, I
> >>>>>> have
> >>>>>>>>>> updated
> >>>>>>>>>>> it
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> documentation.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Some common cache features
> >>> should
> >>>> be
> >>>>>>>>>> implemented
> >>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
> >>>>>> alternative
> >>>>>>> 1
> >>>>>>>>>> turns
> >>>>>>>>>>>> out
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
> >>> this
> >>>>> part
> >>>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> content.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
> >>>> forward,
> >>>>>> the
> >>>>>>>>>> option 2
> >>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
> >>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
> >>>>> offline,
> >>>>>>> we
> >>>>>>>>>> think
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>> 2
> >>>>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
> >>>>>>>> inconsistency
> >>>>>>>>>>>> issues.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
> >>>>> descriptions
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>> content.
> >>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>> to clean them up?
> >>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
> >>>> document
> >>>>>> and
> >>>>>>>>>>>> appreciate
> >>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
> >>> in
> >>>>>>>> `Rejected
> >>>>>>>>>>>>>>> Alternatives`
> >>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
> >>> `Rejected
> >>>>>>>>>> Alternatives`.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best
> >>>>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
> >>> Shammon
> >>>>> FY
> >>>>>> <
> >>>>>>>>>>>>>>> zjureel@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
> >>>>>> `Proposed
> >>>>>>>>>>> Changes`,
> >>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>> put
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
> >>>>>>>> Alternatives`? I
> >>>>>>>>>>> think
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>> help
> >>>>>>>>>>>>>>>>>>>> us
> >>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
> >>> Jing
> >>>> Ge
> >>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
> >>> am
> >>>>>> still
> >>>>>>>>>> confused
> >>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>> completely
> >>>>>>>>>>>>>>>>>>>>>>> reading
> >>>>>>>>>>>>>>>>>>>>>>>> the thread with following
> >>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
> >>>>> registerCatalog()
> >>>>>>> and
> >>>>>>>>>>>>>> addCatalog()
> >>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>>>>>> big
> >>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
> >>>> One
> >>>>> of
> >>>>>>>> them is
> >>>>>>>>>>>>>>> responsible
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
> >>>>>> persistCatalog()?
> >>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
> >>>> Map<String,
> >>>>>>>> Catalog>
> >>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>> as a
> >>>>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
> >>> data
> >>>>>>>> persistence.
> >>>>>>>>>> I
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>> describing
> >>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
> >>>>> clearly
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>> Some
> >>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
> >>>> i.e.
> >>>>>> data
> >>>>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
> >>>> instance
> >>>>>>>> should be
> >>>>>>>>>>>> found
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
> >>>>>> initialized
> >>>>>>>> or it
> >>>>>>>>>>> will
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>>>>> initialized)
> >>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
> >>>>>>> consistency
> >>>>>>>> will
> >>>>>>>>>>> be
> >>>>>>>>>>>>>> taken
> >>>>>>>>>>>>>>>> care
> >>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> while
> >>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
> >>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
> >>>>>> forward,
> >>>>>>>> the
> >>>>>>>>>>>> option
> >>>>>>>>>>>>> 2
> >>>>>>>>>>>>>>>>> solution
> >>>>>>>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
> >>>>>> because,
> >>>>>>>> afaiu,
> >>>>>>>>>>>>> issues
> >>>>>>>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
> >>>>> solved
> >>>>>>>> yet. Do
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> want
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> propose
> >>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
> >>>> suggestions
> >>>>>> for
> >>>>>>>> both
> >>>>>>>>>> of
> >>>>>>>>>>>>> them?
> >>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
> >>>> there
> >>>>>> are
> >>>>>>>> some
> >>>>>>>>>>>>>>> inconsistent
> >>>>>>>>>>>>>>>>>>>>> descriptions
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
> >>>> clean
> >>>>>>> them
> >>>>>>>> up?
> >>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jing
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
> >>> Feng
> >>>>>> Jin <
> >>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
> >>> and
> >>>> I
> >>>>>>>> completely
> >>>>>>>>>>>> agree
> >>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> view.
> >>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
> >>>>>> complicated
> >>>>>>>> the
> >>>>>>>>>>> whole
> >>>>>>>>>>>>>> issue.
> >>>>>>>>>>>>>>>>> First
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> foremost,
> >>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
> >>>> persistence
> >>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> Catalog's
> >>>>>>>>>>>>>>>>>>>> Configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
> >>>>>> persistence
> >>>>>>>> for
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>> Configuration,
> >>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
> >>> to
> >>>> the
> >>>>>>>> Catalog
> >>>>>>>>>>>>>> interface.
> >>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
> >>> Catalog
> >>>>>>>> instance
> >>>>>>>>>> to a
> >>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
> >>>>>>>> implementation
> >>>>>>>>>> will
> >>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
> >>>>>>>>>>>>>>>>>>>>>>>>> /**
> >>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
> >>> the
> >>>>>>>> properties
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>> object.
> >>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
> >>> the
> >>>>>>>> properties
> >>>>>>>>>> of
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>> object
> >>>>>>>>>>>>>>>>>>>>>>>>> * @throws
> >>>>>>>> UnsupportedOperationException if
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> implementing
> >>>>>>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>>>>>> not override
> >>>>>>>>>>>>>>>>>>>>>>>>> * the default
> >>> implementation
> >>>> of
> >>>>>> this
> >>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> */
> >>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
> >>>>>>>> toProperties()
> >>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>> throw new
> >>>>>>>>>>>> UnsupportedOperationException("Please
> >>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>> toProperties
> >>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
> >>>> follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
> >>> configured a
> >>>>>>>>>> CatalogStore,
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> toConfiguration()
> >>>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> will be called when
> >>> registering a
> >>>>>>> Catalog
> >>>>>>>>>>>> instance
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>> catalogName,
> >>>>>>>> Catalog
> >>>>>>>>>>>>> catalog).
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
> >>>> Map<String,
> >>>>>>>> String>
> >>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> saved
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
> >>>>>> instances
> >>>>>>>> do
> >>>>>>>>>> not
> >>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
> >>>>>>>> CatalogStore,
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> toConfiguration()
> >>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
> >>>>>>> consistency
> >>>>>>>> with
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>>> process.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
> >>>>>> Catalog>
> >>>>>>>>>>> catalogs
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
> >>>>>> conflicts
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
> >>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
> >>>>> itself
> >>>>>>> and
> >>>>>>>>>>> provide
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> dedicated
> >>>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
> >>> loading,
> >>>>> such
> >>>>>>> as
> >>>>>>>>>>>>>>>>> ConfigurationCatalog
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> public class
> >>> ConfigurationCatalog
> >>>>>>>> implements
> >>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> ConfigurationCatalog(Map<String,
> >>>>>>> String>
> >>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
> >>> the
> >>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
> >>> 10:03 AM
> >>>>>> Shammon
> >>>>>>>> FY <
> >>>>>>>>>>>>>>>>>> zjureel@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
> >>> the
> >>>>>>>> functionality
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> `CatalogStore`,
> >>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
> >>>> multiple
> >>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
> >>>>>> implementation
> >>>>>>> in
> >>>>>>>>>> Trino
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> same,
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
> >>> only
> >>>>> has
> >>>>>>> one
> >>>>>>>>>>>>>>> `createCatalog`,
> >>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
> >>>>>> `CatalogStore`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
> >>>>>> another
> >>>>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
> >>> `Map<String,
> >>>>>>> Catalog>
> >>>>>>>>>>>> catalogs`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> `CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
> >>>> mentioned
> >>>>>>>> before,
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> `Map<String,
> >>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
> >>>>>>>> `CatalogManager`.
> >>>>>>>>>> How
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>> saving
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
> >>> catalogs`
> >>>>> and
> >>>>>>>>>>>> `CatalogStore
> >>>>>>>>>>>>>>>>>>> catalogStore`
> >>>>>>>>>>>>>>>>>>>>>>>> together
> >>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
> >>> added
> >>>> in
> >>>>>>>>>>>>> `CatalogManager`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>> 5:22 PM
> >>>>> Feng
> >>>>>>>> Jin <
> >>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
> >>> I
> >>>> am
> >>>>>> very
> >>>>>>>> sorry
> >>>>>>>>>>> for
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> misunderstanding
> >>>>>>>>>>>>>>>>>>>>>>>>> caused
> >>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
> >>>>> original
> >>>>>>>>>>> discussion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> >>>>>> pre-discussion
> >>>>>>>> [1]
> >>>>>>>>>> for
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
> >>> such a
> >>>>>>>> discussion
> >>>>>>>>>>>> before.
> >>>>>>>>>>>>>>>>> However,
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> designing
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
> >>> that
> >>>>> the
> >>>>>>>> logic of
> >>>>>>>>>>>>>>>>> CatalogManager
> >>>>>>>>>>>>>>>>>>>> itself
> >>>>>>>>>>>>>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
> >>>> cannot
> >>>>>>> only
> >>>>>>>>>>> persist
> >>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
> >>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
> >>>> Catalogs,
> >>>>>>>> because
> >>>>>>>>>> in
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> end
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> save
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
> >>>>> corresponding
> >>>>>> to
> >>>>>>>> the
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>> instances.
> >>>>>>>>>>>>>>>>>>>>>>>> Therefore,
> >>>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
> >>>>>>> CatalogStore
> >>>>>>>>>>>> interface
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
> >>> this
> >>>>>> part, I
> >>>>>>>> also
> >>>>>>>>>>> took
> >>>>>>>>>>>>>>>>> inspiration
> >>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>> Trino's
> >>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> >>>> handling
> >>>>>>>> catalog
> >>>>>>>>>> with
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>> looks a
> >>>>>>>>>>>>>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> >>>> suggestion. I
> >>>>>>> will
> >>>>>>>>>>> provide
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> detailed
> >>>>>>>>>>>>>>>>>>>>>> description
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
> >>> part,
> >>>>> and
> >>>>>>> add
> >>>>>>>> it
> >>>>>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
> >>>>>>>> `addCatalog`
> >>>>>>>>>>> still
> >>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>> confused
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
> >>>>>> catalog
> >>>>>>> to
> >>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>> renaming
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
> >>> very
> >>>>>>> personal
> >>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> decide
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
> >>>>> intuitive
> >>>>>>> to
> >>>>>>>> me.
> >>>>>>>>>> I
> >>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>> see
> >>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> it.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
> >>>>> statement,
> >>>>>>>> which
> >>>>>>>>>>>> method
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> called?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> >>>>>> `addCatalog`?
> >>>>>>>> I'm
> >>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
> >>>>>> stement.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
> >>>> original
> >>>>>>>> design
> >>>>>>>>>> was
> >>>>>>>>>>> to
> >>>>>>>>>>>>> add
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> directly
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
> >>> would
> >>>>>>> disrupt
> >>>>>>>> the
> >>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>> logic.
> >>>>>>>>>>>>>>>>>>>>>>> Therefore, I
> >>>>>>>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
> >>>>>>>> configuration to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> initialize
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
> >>> the
> >>>>> same
> >>>>>>> time
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
> >>> neccessary
> >>>> to
> >>>>>>>> provide a
> >>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>> implmentation
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> >>>>> `CatalogStoreFactory`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
> >>>>>> otherwise
> >>>>>>> we
> >>>>>>>>>> would
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> introduce
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> additional
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
> >>>> configuration
> >>>>>> for
> >>>>>>>> lazy
> >>>>>>>>>>>>> loading.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> >>>>>>> registration
> >>>>>>>> for
> >>>>>>>>>>>>> catalog.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
> >>>>>>> registerCatalog(String
> >>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
> >>>>>> interface
> >>>>>>>>>> because
> >>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> instance.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
> >>>>>>>> initialization
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
> >>>>>>>> inaccurate. It
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
> >>>>>>>> registration. I
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> title
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>>> 4:27 PM
> >>>>>>>> Shammon
> >>>>>>>>>> FY <
> >>>>>>>>>>>>>>>>>>> zjureel@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
> >>>>>>>> `registerCatalog`
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
> >>>>> method
> >>>>>>>> named
> >>>>>>>>>>>>>> `addCatalog`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
> >>> the
> >>>>>> memory
> >>>>>>>>>> catalog
> >>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>> `cache`,
> >>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> concept
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
> >>>>>> exposed
> >>>>>>> to
> >>>>>>>>>>> users.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
> >>>>>> pre-discussion
> >>>>>>>> [1]
> >>>>>>>>>> for
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>> Please
> >>>>>>>>>>>>>>>>>>>>>>> correct
> >>>>>>>>>>>>>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
> >>>>> conclusion
> >>>>>>> of
> >>>>>>>> that
> >>>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
> >>>>>> interface
> >>>>>>>> and
> >>>>>>>>>>>>> implement
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>>>>> stores
> >>>>>>>>>>>>>>>>>>>>>>>>>>> such
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
> >>>> external
> >>>>>>>> system.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
> >>>>> between
> >>>>>>> the
> >>>>>>>>>>> current
> >>>>>>>>>>>>> FLIP
> >>>>>>>>>>>>>>>>> design
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
> >>> of
> >>>>> the
> >>>>>>>>>> discussion
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>> thread
> >>>>>>>>>>>>>>>>>> [1] ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
> >>>>> 3:41 PM
> >>>>>>>> yuxia <
> >>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
> >>> driving
> >>>>> this
> >>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
> >>>>> handling
> >>>>>>>> catalog
> >>>>>>>>>>> with
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>> looks a
> >>>>>>>>>>>>>>>>>>>>>>>>>> little
> >>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
> >>>>> it'll
> >>>>>> be
> >>>>>>>>>> better
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> java
> >>>>>>>>>>>>>>>>>>>>>>>>> doc
> >>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
> >>> give a
> >>>>>> brief
> >>>>>>>>>> example
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> FLIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
> >>> name
> >>>>>>>> `addCatalog`
> >>>>>>>>>>>> still
> >>>>>>>>>>>>>>> looks
> >>>>>>>>>>>>>>>>>>> confused
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
> >>>> storing
> >>>>>>>> catalog to
> >>>>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>> renaming
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
> >>> It's
> >>>>> very
> >>>>>>>> personal
> >>>>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>> decide
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
> >>>>>>> statement,
> >>>>>>>>>> which
> >>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> called?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
> >>>>>>>> `addCatalog`?
> >>>>>>>>>> I'm
> >>>>>>>>>>>>>>> wondering
> >>>>>>>>>>>>>>>>>>> whether
> >>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>> add a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
> >>> SQL
> >>>>>>>> stement.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
> >>>> neccessary
> >>>>> to
> >>>>>>>>>> provide a
> >>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>> implmentation
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
> >>>>>>> `CatalogStoreFactory`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
> >>>>>>>> registration
> >>>>>>>>>> for
> >>>>>>>>>>>>>> catalog.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
> >>> catalog
> >>>>>> with
> >>>>>>>> CREATE
> >>>>>>>>>>>>>> CATALOG,
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> >>>> used
> >>>>>> by
> >>>>>>>>>> default.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
> >>>>> registration
> >>>>>> is
> >>>>>>>> the
> >>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>> behavior,
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
> >>> to
> >>>>>>>> synchronous
> >>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>> before?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
> >>>>> `addCatalog`
> >>>>>>> and
> >>>>>>>>>>>>>>>> `registerCatalog`
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
> >>>>>>>> registration,
> >>>>>>>>>> it
> >>>>>>>>>>>> may
> >>>>>>>>>>>>>> well
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>>>>>>>>>>> CATALOG
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
> >>>> but
> >>>>>> then
> >>>>>>>> the
> >>>>>>>>>>>>> following
> >>>>>>>>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>>>>>>> TABLE
> >>>>>>>>>>>>>>>>>>>>>>>>> statement
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
> >>> fail
> >>>>> to
> >>>>>>>> open.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
> >>>> change
> >>>>>>> which
> >>>>>>>>>>> should
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> highlighted
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
> >>> part.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
> >>>> asynchronous
> >>>>>>>>>>>> registration, I
> >>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> expect
> >>>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
> >>>> open
> >>>>>> or
> >>>>>>>>>> register
> >>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> background,
> >>>>>>>>>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
> >>> comments,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
> >>>> will
> >>>>> be
> >>>>>>>>>>> initialized
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
> >>>> initialization
> >>>>>>>> process
> >>>>>>>>>>> fails,
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>> statements
> >>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
> >>> successfully."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
> >>> lazy
> >>>>>>>>>> initialization
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
> >>>>>>>>>> jinfeng1995@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
> >>>>>>> dev@flink.apache.org
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
> >>> 03日
> >>>> 下午
> >>>>>>>> 3:27:45
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
> >>>> 295:
> >>>>>>>> Support
> >>>>>>>>>>>>>> persistence
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
> >>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
> >>>> all
> >>>>>> for
> >>>>>>>> your
> >>>>>>>>>>>>> interest
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> DISCUSS.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
> >>> catalog
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>>> same
> >>>>>>>>>>>>> name
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
> >>>> crucial
> >>>>>>>> point.
> >>>>>>>>>>> Based
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>> current
> >>>>>>>>>>>>>>>>>>>>>> personal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
> >>>>>> Map<String,
> >>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> serve
> >>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> cache
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
> >>> catalogs
> >>>>> and
> >>>>>>>> have
> >>>>>>>>>> the
> >>>>>>>>>>>>>> highest
> >>>>>>>>>>>>>>>>>>> priority.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
> >>> methods
> >>>>> that
> >>>>>>> can
> >>>>>>>>>> affect
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> >>> registerCatalog(String
> >>>>>>>>>> catalogName,
> >>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
> >>>>> catalog
> >>>>>>>> instance
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>>> unregisterCatalog(String
> >>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> >>> the
> >>>>>> catalog
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogName
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
> >>>>> Catalog>
> >>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
> >>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
> >>>> retrieves
> >>>>>> the
> >>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
> >>> Catalog>
> >>>>>>>> catalogs. If
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>> exist,
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
> >>>> corresponding
> >>>>>>>>>>> configuration
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
> >>> puts
> >>>>> the
> >>>>>>>>>>> initialized
> >>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> >>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
> >>> methods
> >>>>>> only
> >>>>>>>> modify
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
> >>>>>>> catalogName,
> >>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
> >>>>>>> properties
> >>>>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> catalogStore
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> checks
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
> >>>>>> catalogName
> >>>>>>>> with
> >>>>>>>>>> the
> >>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>> name.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>> removeCatalog(String
> >>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
> >>> the
> >>>>>>> specified
> >>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> specified
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
> >>>>>>> catalogStore.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
> >>>> possible
> >>>>>>>> conflict
> >>>>>>>>>>>>>> scenarios:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
> >>> in
> >>>> the
> >>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>> Catalog>,
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>>>>>> catalogName,
> >>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
> >>> directly
> >>>>> saved
> >>>>>>> to
> >>>>>>>> the
> >>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
> >>>>>>> Map<String,
> >>>>>>>>>>>> Catalog>,
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
> >>>>> catalog)
> >>>>>>>> method
> >>>>>>>>>>> will
> >>>>>>>>>>>>>> fail.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
> >>> corresponding
> >>>>>>>> catalogName
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> exists
> >>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
> >>> the
> >>>>>>>>>>>> addCatalog(String
> >>>>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
> >>>> method
> >>>>>> can
> >>>>>>>>>> directly
> >>>>>>>>>>>>> save
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
> >>>>>>>>>> getCatalog(String
> >>>>>>>>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
> >>>> for
> >>>>>>>>>>> initialization
> >>>>>>>>>>>>>>> because
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> corresponding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
> >>> already
> >>>>>> exists
> >>>>>>>> in
> >>>>>>>>>>>> catalogs
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
> >>>>>>>>>>>>> unregisterCatalog(String
> >>>>>>>>>>>>>>>>>>>> catalogName)
> >>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
> >>>>>>>> corresponding to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> original
> >>>>>>>>>>>>>>>>>>>>> catalogName
> >>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
> >>> confuse
> >>>>>> users
> >>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
> >>>>> catalog)`
> >>>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>>>> `Map<String,
> >>>>>>>>>>>>>>>>>>>> Catalog>
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> `registerCatalog(String
> >>>>>>>> catalogName,
> >>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>> String>
> >>>>>>>>>>>>>>>>>>>>>>>>> properties)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
> >>>> lead
> >>>>>> to
> >>>>>>>>>>>> confusion. I
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>> changing
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
> >>> to
> >>>>>>>>>>> addCatalog(String
> >>>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>>>>> Map<String,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
> >>> as
> >>>>>>>> previously
> >>>>>>>>>>>>> mentioned
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> >>>>> `registerCatalog(String
> >>>>>>>>>>>>>>> catalogName,Catalog
> >>>>>>>>>>>>>>>>>>>> catalog,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
> >>> method
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
> >>>> already
> >>>>> an
> >>>>>>>>>> instance,
> >>>>>>>>>>>>>> adding
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> "lazyInit"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
> >>>>>>>>>> registerCatalog(String
> >>>>>>>>>>>>>>>>> catalogName,
> >>>>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>>> catalog)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
> >>>> necessarily
> >>>>>>>> result in
> >>>>>>>>>>> lazy
> >>>>>>>>>>>>>>>>>>> initialization.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
> >>>> about
> >>>>>>>>>> encryption
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
> >>>>>>> necessary,
> >>>>>>>> but
> >>>>>>>>>>>>> perhaps
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> encryption
> >>>>>>>>>>>>>>>>>>>>>>> logic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
> >>> in
> >>>>> the
> >>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
> >>> the
> >>>>>>>>>>> FileCatalogStore
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>> encrypt
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
> >>>>> while
> >>>>>>> the
> >>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>> require
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
> >>> the
> >>>>>>>>>>>>>> `MemoryCatalogStore`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
> >>> necessary
> >>>> to
> >>>>>>> have a
> >>>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
> >>>> saves
> >>>>>>>> Catalog
> >>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> memory.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
> >>> to
> >>>>>>>> implement
> >>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>> loading
> >>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> Catalog,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
> >>> introduce
> >>>>>>>> additional
> >>>>>>>>>>> cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
> >>>>>>>> registration
> >>>>>>>>>>>> failed?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
> >>> critical
> >>>>>>> concern.
> >>>>>>>>>> When
> >>>>>>>>>>>>>>> executing
> >>>>>>>>>>>>>>>>> DDL,
> >>>>>>>>>>>>>>>>>>>> DQL,
> >>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>> DML
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
> >>>> reference a
> >>>>>>>> specific
> >>>>>>>>>>>>> Catalog,
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> Catalog
> >>>>>>>>>>>>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
> >>> if it
> >>>>> has
> >>>>>>> not
> >>>>>>>>>> been
> >>>>>>>>>>>>>>>> initialized
> >>>>>>>>>>>>>>>>>> yet.
> >>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
> >>>>> fails,
> >>>>>>>> these
> >>>>>>>>>>>>>> statements
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> executed
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
> >>>>>> Catalog>
> >>>>>>>>>> catalogs
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
> >>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
> >>>> have
> >>>>>> the
> >>>>>>>> same
> >>>>>>>>>>>>> catalog
> >>>>>>>>>>>>>>>> name,
> >>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>>>> correponding
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> >>>>> instances.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
> >>>> resolved
> >>>>> by
> >>>>>>>>>> referring
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>>>>>>>>> responses.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
> >>> the
> >>>>>>> Catalog
> >>>>>>>>>> that
> >>>>>>>>>>>> has
> >>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>> recently
> >>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
> >>>> for
> >>>>>>>> subsequent
> >>>>>>>>>>>> use.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
> >>>>>> 10:58 PM
> >>>>>>>>>> Xianxun
> >>>>>>>>>>>> Ye
> >>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
> >>> this
> >>>>>> Flip,
> >>>>>>>> I do
> >>>>>>>>>>>>> believe
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> Flip
> >>>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> helpful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
> >>>> work
> >>>>>> for
> >>>>>>>> also
> >>>>>>>>>>>>> manages a
> >>>>>>>>>>>>>>> lot
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> catalogs,
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
> >>>> becomes
> >>>>>>> slow
> >>>>>>>>>>> because
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> loading a
> >>>>>>>>>>>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
> >>> catalogs in
> >>>>> the
> >>>>>>>> user's
> >>>>>>>>>>>> Flink
> >>>>>>>>>>>>>> SQL
> >>>>>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>>>>>>>>> regular
> >>>>>>>>>>>>>>>>>>>>>>>>>>> expression
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
> >>>>> catalogs
> >>>>>>> to
> >>>>>>>>>>> improve
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> speed
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>>>>>>> submission.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
> >>> flip-295, I
> >>>>>> have
> >>>>>>>> some
> >>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
> >>>>> catalog
> >>>>>>> with
> >>>>>>>>>>> CREATE
> >>>>>>>>>>>>>>> CATALOG,
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
> >>>> used
> >>>>>> by
> >>>>>>>>>> default.
> >>>>>>>>>>>>> What
> >>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>>>>>>>>>>> registration
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
> >>>>> implement
> >>>>>>>>>>> asynchronous
> >>>>>>>>>>>>>>>>>> registration?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
> >>> same
> >>>>>>>> question
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> Shammon’s
> >>>>>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>>>>>>>> comment.
> >>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
> >>>>> catalogs
> >>>>>>> and
> >>>>>>>>>>>>>> CatalogStore
> >>>>>>>>>>>>>>>>>>>> catalogstore
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
> >>> have
> >>>> the
> >>>>>>> same
> >>>>>>>>>>> catalog
> >>>>>>>>>>>>>> name,
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>>> correponding
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
> >>>>> instances.
> >>>>>>> For
> >>>>>>>>>>> example:
> >>>>>>>>>>>>>>>>> catalog1->
> >>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
> >>> catalog1 ->
> >>>>>>>> hiveCatalog
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> catalogstore.
> >>>>>>>>>>>>>>>>>>>>>>> Would
> >>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> case
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
> >>> should
> >>>>>> define
> >>>>>>>>>>> `catalogs`
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `catalogstore`
> >>>>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
> >>> 11:13，Hang
> >>>>>> Ruan
> >>>>>>> <
> >>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 写道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Leonard Xu <xb...@gmail.com>.
Hi, Feng
Sorry for reply late, but I’ve some comments about the FLIP
1. The introduced Public class CatalogDescriptor seems missed some necessary component
a) How to construct a CatalogDescriptor ?
b) How to visit the fields ? Could we use Configuration instead of Map<String, String> ?
c) Could we offer a built-in factory method to build a CatalogDescriptor like
public static CatalogDescriptor of(String catalogName, Configuration configuration)
2. The FLIP said “By default, there are two built-in CatalogStores available: the In-Memory CatalogStore and the File CatalogStore” ,
Do we have plan to offer a default CatalogStore if user didn’t config this? IIUC, users can obtains the benefits from lazy catalog initialization If
we have a default catalogstore even it is in-memory.
Best,
Leonard
> On Jun 6, 2023, at 7:08 PM, Feng Jin <ji...@gmail.com> wrote:
>
> Hi everyone,
>
> Thank you everyone for your valuable input. If there are no further
> questions or concerns about the FLIP[1], I would like to start voting
> tomorrow (6/7).
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
>
> Best,
> Feng
>
>
> On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
>
>> Hi Samrat,
>>
>> Thanks for your advice.
>>
>>> 1. The createCatalog method does not mention any exceptions being
>> thrown.
>>
>> CreateCatalog will throw CatalogException like registerCatalog. As
>> CatalogException is a RuntimeException,
>> there is no explicit declaration of throwing Exceptions in CatalogManager
>> and TableEnvironment.
>> To avoid misunderstandings, I have added the "throw CatalogException" flag
>> to the createCatalog method definition of CatalogStore.
>>
>>> 2. Could you please provide an exhaustive list of the supported kinds?
>>
>> Sure, the documentation now includes the configuration of the built-in
>> CatalogStore as well as how to configure a custom CatalogStore.
>>
>>
>> Best,
>> Feng
>>
>>
>> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com> wrote:
>>
>>> Hi Feng,
>>>
>>> Thank you for providing the proposal. I believe this feature will be
>>> highly
>>> valuable.
>>>
>>> I have a couple of inquiries:
>>>
>>> 1. According to the documentation [1], the createCatalog method does not
>>> mention any exceptions being thrown. However, I would like to confirm if
>>> it
>>> is always true that there will be no failures in all scenarios. Please let
>>> me know if there is any additional information I may have missed.
>>>
>>> 2. Regarding the registration process using the `table.catalog-store.kind`
>>> configuration, could you please provide an exhaustive list of the
>>> supported
>>> kinds?
>>> It would be great to have a comprehensive understanding of the options
>>> available.
>>>
>>> Bests,
>>> Samrat
>>>
>>> [1]
>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>>
>>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
>>>
>>>> Hi Hang and Jark
>>>>
>>>> Thank you very much for your reply.
>>>>
>>>> @Jark
>>>>> 1. Could you move the CatalogStore registration API to the "Public
>>>> Interface" section?
>>>>
>>>> Indeed, it is more reasonable.
>>>>
>>>>> 2. We should prefix "table." for the CatalogStore configuration.
>>>>
>>>> Sure, the names that you provided are indeed more logical and easier
>>> to
>>>> understand.
>>>>
>>>>> 3. About the open/close method in CatalogStoreFactory.
>>>>
>>>> I also agree with the proposed requirement scenario and design. I have
>>>> already made modifications to the interface.
>>>>
>>>> @Hang
>>>>
>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or close
>>>> the resource.
>>>>
>>>> I have already added the missing method.
>>>>
>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>>>>
>>>> Sorry for the misspelling, I updated this part.
>>>>
>>>>> 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
>>>> to init or close its resource
>>>>
>>>> This is indeed a scenario that needs to be considered, and I have added
>>>> relevant methods accordingly.
>>>> Additionally, I have included some relevant description information in
>>> the
>>>> documentation to help others understand
>>>> why the open/close methods need to be added.
>>>>
>>>>
>>>> Best,
>>>> Feng
>>>>
>>>> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
>>>>
>>>>> Hi Jing,
>>>>>
>>>>> Thank you for the update.
>>>>>
>>>>> 1. Could you move the CatalogStore registration API to the "Public
>>>>> Interface" section?
>>>>> "Proposed Changes" is more like a place to describe the implementation
>>>>> details.
>>>>>
>>>>> 2. We should prefix "table." for the CatalogStore configuration.
>>> Besides,
>>>>> the config key
>>>>> name should be hierarchical[1]. Therefore, it may be better to use:
>>>>> "table.catalog-store.kind"
>>>>> "table.catalog-store.file.path"
>>>>>
>>>>> 3. I think Hang's suggestions make sense.
>>>>>
>>>>> Others look good to me.
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Hi, Feng.
>>>>>>
>>>>>> Thanks for the update.
>>>>>> The current design looks good to me. I have some minor comments.
>>>>>>
>>>>>> 1. The `CatalogStore` need the `open`/`close` methods to init or
>>> close
>>>>> the
>>>>>> resource. For example, when storing the information in MySQL, the
>>> store
>>>>>> needs to open and close the connections.
>>>>>>
>>>>>> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>>>>>>
>>>>>> 3. About the usage in the sql gateway.
>>>>>> Considering the usage in sql gateway, the sql gateway may create a
>>>>>> CatalogStore for each session.
>>>>>> If we are using the MySqlCatalogStore, there would be so many
>>>>> connections.
>>>>>> How can we reuse the connection among these sessions?
>>>>>> I think sql gateway need to maintain a connection pool in
>>>>>> the CatalogStoreFactory and each session get its own connection from
>>>> the
>>>>>> pool when it is created.
>>>>>> Then the `CatalogStoreFactory` may need the `open`/`close` methods
>>> to
>>>>> init
>>>>>> or close its resource.
>>>>>> Or is there a better way?
>>>>>>
>>>>>> Best,
>>>>>> Hang
>>>>>>
>>>>>> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>>>>>>
>>>>>>> Thanks Jingsong.
>>>>>>>
>>>>>>>> Just naming, maybe `createCatalog` in TableEnv
>>>>>>>
>>>>>>> +1 For this, I have already updated FLIP.
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Feng
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
>>> jingsonglee0@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Feng,
>>>>>>>>
>>>>>>>> Just naming, maybe `createCatalog` in TableEnv, I can see many
>>>>>>>> functions are converted to createxxx from registerxxx.
>>>>>>>>
>>>>>>>> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi jark, thanks for your suggestion.
>>>>>>>>>
>>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>>> the
>>>>>>>>> CatalogStore should be immutable once TableEnv is created.
>>>>> Otherwise,
>>>>>>>> there
>>>>>>>>> might be some data inconsistencies when CatalogStore is
>>> changed.
>>>>>>>>>
>>>>>>>>> Yes, We should initialize the CatalogStore when creating the
>>>>>> TableEnv.
>>>>>>>>> Therefore, my current proposal is to add a method to configure
>>>> the
>>>>>>>>> CatalogStore in EnvironmentSettings.
>>>>>>>>>
>>>>>>>>> // Initialize a catalog Store
>>>>>>>>> CatalogStore catalogStore = new FileCatalogStore("");
>>>>>>>>>
>>>>>>>>> // set up the Table API
>>>>>>>>> final EnvironmentSettings settings =
>>>>>>>>>
>>> EnvironmentSettings.newInstance().inBatchMode()
>>>>>>>>> .withCatalogStore(catalogStore)
>>>>>>>>> .build();
>>>>>>>>>
>>>>>>>>> final TableEnvironment tableEnv =
>>>>>>>> TableEnvironment.create(settings);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>>>> default
>>>>>>>> method,
>>>>>>>>> not an interface method?
>>>>>>>>>
>>>>>>>>> Sorry, While I did refer to the Catalog interface, I agree
>>> that
>>>> as
>>>>> a
>>>>>>> new
>>>>>>>>> interface, the CatalogStoreFactory should not have a default
>>>> method
>>>>>> but
>>>>>>>> an
>>>>>>>>> interface method. I have already modified the interface.
>>>>>>>>>
>>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>>>>>>>>> deprecated`registerCatalog`.
>>>>>>>>>> 4. In the "Compatibility" section, would be better to
>>> mention
>>>> the
>>>>>>>> changed
>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
>>>> other
>>>>>>>>> persisted catalog store) is used.
>>>>>>>>>
>>>>>>>>> Thanks for the suggestion, I have updated the FLIP [1].
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1].
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Feng
>>>>>>>>>
>>>>>>>>> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Feng,
>>>>>>>>>>
>>>>>>>>>> This is a useful FLIP. Thanks for starting this discussion.
>>>>>>>>>> The current design looks pretty good to me. I just have some
>>>>> minor
>>>>>>>>>> comments.
>>>>>>>>>>
>>>>>>>>>> 1. How to register the CatalogStore for Table API? I think
>>> the
>>>>>>>> CatalogStore
>>>>>>>>>> should be immutable once TableEnv is created. Otherwise,
>>> there
>>>>>> might
>>>>>>> be
>>>>>>>>>> some data inconsistencies when CatalogStore is changed.
>>>>>>>>>>
>>>>>>>>>> 2. Why does the CatalogStoreFactory interface only have a
>>>> default
>>>>>>>> method,
>>>>>>>>>> not an interface method?
>>>>>>>>>>
>>>>>>>>>> 3. Please mention the alternative API in Javadoc for the
>>>>> deprecated
>>>>>>>>>> `registerCatalog`.
>>>>>>>>>>
>>>>>>>>>> 4. In the "Compatibility" section, would be better to
>>> mention
>>>> the
>>>>>>>> changed
>>>>>>>>>> behavior of CREATE CATALOG statement if FileCatalogStore (or
>>>>> other
>>>>>>>>>> persisted catalog store) is used.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jark
>>>>>>>>>>
>>>>>>>>>> On Thu, 1 Jun 2023 at 11:26, Feng Jin <
>>> jinfeng1995@gmail.com>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi ， thanks all for reviewing the flip.
>>>>>>>>>>>
>>>>>>>>>>> @Ron
>>>>>>>>>>>
>>>>>>>>>>>> Regarding the CatalogStoreFactory#createCatalogStore
>>>> method,
>>>>>> do
>>>>>>> we
>>>>>>>>>> need
>>>>>>>>>>> to provide a default implementation?
>>>>>>>>>>>
>>>>>>>>>>> Yes, we will provide a default
>>> InMemoryCatalogStoreFactory to
>>>>>>> create
>>>>>>>> an
>>>>>>>>>>> InMemoryCatalogStore.
>>>>>>>>>>>
>>>>>>>>>>>> If we get a Catalog from CatalogStore, after
>>> initializing
>>>>> it,
>>>>>>>> whether
>>>>>>>>>> we
>>>>>>>>>>> put it in Map<String, Catalog> catalogs again?
>>>>>>>>>>>
>>>>>>>>>>> Yes, in the current design, catalogs are stored as
>>> snapshots,
>>>>> and
>>>>>>>> once
>>>>>>>>>>> initialized, the Catalog will be placed in the Map<String,
>>>>>> Catalog>
>>>>>>>>>>> catalogs.
>>>>>>>>>>> Subsequently, the Map<String, Catalog> catalogs will be
>>> the
>>>>>> primary
>>>>>>>>>> source
>>>>>>>>>>> for obtaining the corresponding Catalog.
>>>>>>>>>>>
>>>>>>>>>>>> how about renaming them to `catalog.store.type` and
>>>>>>>>>>> `catalog.store.path`?
>>>>>>>>>>>
>>>>>>>>>>> I think it is okay. Adding "sql" at the beginning may
>>> seem a
>>>>> bit
>>>>>>>>>> strange. I
>>>>>>>>>>> will update the FLIP.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> @Shammon
>>>>>>>>>>>
>>>>>>>>>>> Thank you for the review. I have made the necessary
>>>>> corrections.
>>>>>>>>>>> Regarding the modifications made to the Public Interface,
>>> I
>>>>> have
>>>>>>> also
>>>>>>>>>>> included the relevant changes to the `TableEnvironment`.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Feng
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 31, 2023 at 5:02 PM Shammon FY <
>>>> zjureel@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi feng,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for updating, I have some minor comments
>>>>>>>>>>>>
>>>>>>>>>>>> 1. The modification of `CatalogManager` should not be in
>>>>>> `Public
>>>>>>>>>>>> Interfaces`, it is not a public interface.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. `@PublicEvolving` should be added for `CatalogStore`
>>> and
>>>>>>>>>>>> `CatalogStoreFactory`
>>>>>>>>>>>>
>>>>>>>>>>>> 3. The code `Optional<Catalog> optionalDescriptor =
>>>>>>>>>>>> catalogStore.get(catalogName);` in the `CatalogManager`
>>>>> should
>>>>>> be
>>>>>>>>>>>> `Optional<CatalogDescriptor> optionalDescriptor =
>>>>>>>>>>>> catalogStore.get(catalogName);`
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 31, 2023 at 2:24 PM liu ron <
>>>> ron9.liu@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, Feng
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for driving this FLIP, this proposal is very
>>>> useful
>>>>>> for
>>>>>>>>>> catalog
>>>>>>>>>>>>> management.
>>>>>>>>>>>>> I have some small questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Regarding the
>>> CatalogStoreFactory#createCatalogStore
>>>>>> method,
>>>>>>>> do we
>>>>>>>>>>>> need
>>>>>>>>>>>>> to provide a default implementation?
>>>>>>>>>>>>> 2. If we get Catalog from CatalogStore, after
>>>> initializing
>>>>>> it,
>>>>>>>>>> whether
>>>>>>>>>>> we
>>>>>>>>>>>>> put it to Map<String, Catalog> catalogs again?
>>>>>>>>>>>>> 3. Regarding the options `sql.catalog.store.type` and
>>>>>>>>>>>>> `sql.catalog.store.file.path`, how about renaming
>>> them to
>>>>>>>>>>>>> `catalog.store.type` and `catalog.store.path`?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>
>>>>>>>>>>>>> Feng Jin <ji...@gmail.com> 于2023年5月29日周一
>>> 21:19写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi yuxia
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But from the code in Proposed Changes, once we
>>>>> register
>>>>>>> the
>>>>>>>>>>> Catalog,
>>>>>>>>>>>>> we
>>>>>>>>>>>>>> initialize it and open it. right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, In order to avoid inconsistent semantics of the
>>>>>> original
>>>>>>>>>> CREATE
>>>>>>>>>>>>>> CATALOG DDL, Catalog will be directly initialized in
>>>>>>>>>> registerCatalog
>>>>>>>>>>> so
>>>>>>>>>>>>>> that parameter validation can be performed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the current design, lazy initialization is mainly
>>>>>>> reflected
>>>>>>>> in
>>>>>>>>>>>>>> getCatalog. If CatalogStore has already saved some
>>>>> catalog
>>>>>>>>>>>>> configurations,
>>>>>>>>>>>>>> only initialization is required in getCatalog.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, May 29, 2023 at 8:27 PM yuxia <
>>>>>>>> luoyuxia@alumni.sjtu.edu.cn
>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, Feng.
>>>>>>>>>>>>>>> I'm trying to understanding the meaning of *lazy
>>>>>>>> initialization*.
>>>>>>>>>>> If
>>>>>>>>>>>>> i'm
>>>>>>>>>>>>>>> wrong, please correct me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IIUC, lazy initialization means only you need to
>>>> access
>>>>>> the
>>>>>>>>>>> catalog,
>>>>>>>>>>>>> then
>>>>>>>>>>>>>>> you initialize it. But from the code in Proposed
>>>>> Changes,
>>>>>>>> once we
>>>>>>>>>>>>>> register
>>>>>>>>>>>>>>> the Catalog,
>>>>>>>>>>>>>>> we initialize it and open it. right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Yuxia
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- 原始邮件 -----
>>>>>>>>>>>>>>> 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>>>>>>>>>>>>>>> 收件人: "dev" <de...@flink.apache.org>
>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP 295: Support persistence of
>>>>>> Catalog
>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>> and asynchronous registration
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for your effort! +1 for the proposal.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One of the major changes is that current design
>>> will
>>>>>>> provide
>>>>>>>>>>>>>>> Map<String,Catalog> catalogs as a snapshot instead
>>>> of a
>>>>>>>> cache,
>>>>>>>>>>> which
>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>> once it has been initialized, any changes done by
>>>> other
>>>>>>>> sessions
>>>>>>>>>>> will
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> affect it. Point 6 described follow-up options for
>>>>>> further
>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, May 29, 2023 at 5:31 AM Feng Jin <
>>>>>>>> jinfeng1995@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all, I would like to update you on the latest
>>>>>> progress
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Last week, Leonard Xu, HangRuan, Jing Ge,
>>> Shammon
>>>> FY,
>>>>>>>> ShengKai
>>>>>>>>>>> Fang
>>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>>> had an offline discussion regarding the overall
>>>>>> solution
>>>>>>>> for
>>>>>>>>>>> Flink
>>>>>>>>>>>>>>>> CatalogStore. We have reached a consensus and I
>>>> have
>>>>>>>> updated
>>>>>>>>>> the
>>>>>>>>>>>>> final
>>>>>>>>>>>>>>>> solution in FLIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Next, let me briefly describe the entire design:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Introduce CatalogDescriptor to store catalog
>>>>>>>> configuration
>>>>>>>>>>>> similar
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> TableDescriptor.
>>>>>>>>>>>>>>>> 2.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The two key functions of CatalogStore - void
>>>>>>>>>>> storeCatalog(String
>>>>>>>>>>>>>>>> catalogName, CatalogDescriptor) and
>>>>>> CatalogDescriptor
>>>>>>>>>>>>>>> getCatalog(String)
>>>>>>>>>>>>>>>> will both use CatalogDescriptor instead of
>>>> Catalog
>>>>>>>> instance.
>>>>>>>>>>>> This
>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>> CatalogStore will only be responsible for
>>> saving
>>>>> and
>>>>>>>>>>> retrieving
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> configurations without having to initialize
>>>>>> catalogs.
>>>>>>>>>>>>>>>> 3.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The default registerCatalog(String
>>> catalogName,
>>>>>>> Catalog
>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>> function in CatalogManager will be marked as
>>>>>>> deprecated.
>>>>>>>>>>>>>>>> 4.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A new function registerCatalog(String
>>>> catalogName,
>>>>>>>>>>>>> CatalogDescriptor
>>>>>>>>>>>>>>>> catalog) will be added to serve as the
>>> default
>>>>>>>> registration
>>>>>>>>>>>>> function
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> catalogs in CatalogManager.
>>>>>>>>>>>>>>>> 5.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Map<String,Catalog> catalogs in CataloManager
>>>> will
>>>>>>>> remain
>>>>>>>>>>>>> unchanged
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> save initialized catalogs.This means that
>>>> deletion
>>>>>>>>>> operations
>>>>>>>>>>>> from
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>> session won't synchronize with other
>>> sessions.
>>>>>>>>>>>>>>>> 6.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To support multi-session synchronization
>>>> scenarios
>>>>>> for
>>>>>>>>>>> deletions
>>>>>>>>>>>>>> later
>>>>>>>>>>>>>>>> on we should make Map<String,Catalog>catalogs
>>>>>>>>>>> configurable.There
>>>>>>>>>>>>> may
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> three possible situations:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> a.Default caching of all initialized catalogs
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> b.Introduction of LRU cache logic which can
>>>>>>>> automatically
>>>>>>>>>>> clear
>>>>>>>>>>>>>>>> long-unused catalogs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> c.No caching of any instances; each call to
>>>>>> getCatalog
>>>>>>>>>>> creates a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> instance.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the document for discussion:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the final proposal document:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you very much for your attention and
>>>>> suggestions
>>>>>> on
>>>>>>>> this
>>>>>>>>>>>>> FLIP. A
>>>>>>>>>>>>>>>> special thanks to Hang Ruan for his suggestions
>>> on
>>>>> the
>>>>>>>> entire
>>>>>>>>>>>> design
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> organizing offline discussions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If you have any further suggestions or feedback
>>>> about
>>>>>>> this
>>>>>>>> FLIP
>>>>>>>>>>>>> please
>>>>>>>>>>>>>>> feel
>>>>>>>>>>>>>>>> free to share.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sat, May 6, 2023 at 8:32 PM Jing Ge
>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for improving the FLIP. It looks good
>>> to
>>>> me.
>>>>>> We
>>>>>>>> could
>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>> reconsider in the future how to provide more
>>>> common
>>>>>>>> built-in
>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>> functionality in CatalogManager so that not
>>> every
>>>>>>>>>> CatalogSotre
>>>>>>>>>>>>>>>>> implementation has to take care of it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, May 4, 2023 at 1:47 PM Feng Jin <
>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Jing,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for your reply.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>>>>> suggest
>>>>>>> you
>>>>>>>>>>>>> completely
>>>>>>>>>>>>>>> walk
>>>>>>>>>>>>>>>>>> through the FLIP again and fix those issues
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am very sorry for my carelessness and at
>>> the
>>>>> same
>>>>>>>> time, I
>>>>>>>>>>>>> greatly
>>>>>>>>>>>>>>>>>> appreciate your careful review. I have
>>>> thoroughly
>>>>>>>> checked
>>>>>>>>>> the
>>>>>>>>>>>>>> entire
>>>>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>>> and made corrections to these issues.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If I am not mistaken, with the current
>>> FLIP
>>>>>>> design,
>>>>>>>>>>>>>>> CatalogManager
>>>>>>>>>>>>>>>>>> could work without Optional CatalogStore
>>> being
>>>>>>>> configured.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, in the original design, CatalogStore
>>> was
>>>> not
>>>>>>>> necessary
>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>>> CatalogManager used Map<String, Catalog>
>>>> catalogs
>>>>>> to
>>>>>>>> store
>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>> instances.
>>>>>>>>>>>>>>>>>> However, this caused inconsistency issues.
>>>>>>> Therefore, I
>>>>>>>>>>>> modified
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>> of the design and removed Map<String,
>>> Catalog>
>>>>>>> catalogs
>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> CatalogManager.
>>>>>>>>>>>>>>>>>> At the same time, InMemoryCatalog will
>>> serve
>>>> as
>>>>>> the
>>>>>>>>>> default
>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>> to save catalogs in memory and replace the
>>>>>>>> functionality of
>>>>>>>>>>>>>>>>>> Map<String,Catalog>catalogs.
>>>>>>>>>>>>>>>>>> The previous plan that kept
>>>>>>>> Map<String,Catalog>catalogs has
>>>>>>>>>>>> been
>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> Rejected Alternatives.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
>>>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There are still many places contain
>>>>> inconsistent
>>>>>>>> content,
>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. "Asynchronous registration" is still
>>> used.
>>>>>>>>>>>>>>>>>>> 2. The java comment of the method
>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>> Catalog catalog) in CatalogManager does
>>> not
>>>>> tell
>>>>>>>> what the
>>>>>>>>>>>>> method
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> doing.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> There might be more such issues. I would
>>>>> suggest
>>>>>>> you
>>>>>>>>>>>> completely
>>>>>>>>>>>>>>> walk
>>>>>>>>>>>>>>>>>>> through the FLIP again and fix those
>>> issues.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> About the inMemoryCatalogStore, do you
>>> mean
>>>>> that
>>>>>>> you
>>>>>>>> will
>>>>>>>>>>>> build
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>> functionality in the CatalogStore? This
>>> is a
>>>>> very
>>>>>>>>>> different
>>>>>>>>>>>>>> design
>>>>>>>>>>>>>>>>>> concept
>>>>>>>>>>>>>>>>>>> from what the current FLIP described. If
>>> I am
>>>>> not
>>>>>>>>>> mistaken,
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> current FLIP design, CatalogManager could
>>>> work
>>>>>>>> without
>>>>>>>>>>>> Optional
>>>>>>>>>>>>>>>>>>> CatalogStore being configured. That is the
>>>>> reason
>>>>>>>> why I
>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> last email that the example code wrt the
>>>>> Optional
>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> correct.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
>>>>>>>>>>>>> jinfeng1995@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Jing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> There are still some NIT issues in the
>>>> FLIP
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you very much for the careful
>>>> review. I
>>>>>>> have
>>>>>>>>>>> already
>>>>>>>>>>>>> made
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> relevant changes.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in
>>> the
>>>>>>>>>> multi-instance
>>>>>>>>>>>>>>>> scenarios, I
>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> sure if this is the intended behaviour
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Currently, there are conflicts in
>>> multiple
>>>>>>>> scenarios
>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>> design. I am thinking whether we should
>>>>> remove
>>>>>>>>>>> 'Map<String,
>>>>>>>>>>>>>>>> Catalog>'
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> make Cache the default behavior of
>>>>>>>>>> InMemoryCatalogStore.
>>>>>>>>>>>> This
>>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>> can implement their own CatalogStore to
>>>>> achieve
>>>>>>>>>>>>> multi-instance
>>>>>>>>>>>>>>>>>>>> non-conflicting scenarios. What do you
>>>> think?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
>>>>>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for working on the FLIP. There
>>> are
>>>>>> still
>>>>>>>> some
>>>>>>>>>>> NIT
>>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>>>>>> like:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1. Optional<CatalogStore> catalogStore
>>>> has
>>>>>> been
>>>>>>>> used
>>>>>>>>>> as
>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>>> instead of Optional in the code
>>> example.
>>>> It
>>>>>>>> should be
>>>>>>>>>>>> fine
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>> pseudo code for now and update it
>>> after
>>>> you
>>>>>>>> submit
>>>>>>>>>> the
>>>>>>>>>>>> PR.
>>>>>>>>>>>>>>>>>>>>> 2. addCatalog(...) is still used
>>>> somewhere
>>>>> in
>>>>>>> the
>>>>>>>>>>>> rejected
>>>>>>>>>>>>>>>> section
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>> should be persistContext(...) to keep
>>> it
>>>>>>>> consistent.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Speaking of the conflict issues in the
>>>>>>>> multi-instance
>>>>>>>>>>>>>>> scenarios,
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> sure if this is the intended
>>> behaviour.
>>>> If
>>>>>>>>>> Map<String,
>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> used as a cache, it should be invalid,
>>>> once
>>>>>> the
>>>>>>>>>> related
>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>> removed from the CatalogStore by
>>> another
>>>>>>>> instance.
>>>>>>>>>> Did
>>>>>>>>>>> I
>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>>> something?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:40 PM Feng
>>> Jin
>>>> <
>>>>>>>>>>>>>>> jinfeng1995@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Jing,Shammon
>>>>>>>>>>>>>>>>>>>>>> Thanks for your reply.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> @Jing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> How about persistCatalog()?
>>>>>>>>>>>>>>>>>>>>>> I think this is a good function
>>> name, I
>>>>>> have
>>>>>>>>>> updated
>>>>>>>>>>> it
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Some common cache features
>>> should
>>>> be
>>>>>>>>>> implemented
>>>>>>>>>>>>>>>>>>>>>> Thank you for the suggestion. If
>>>>>> alternative
>>>>>>> 1
>>>>>>>>>> turns
>>>>>>>>>>>> out
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> appropriate later, I will improve
>>> this
>>>>> part
>>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> content.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> As the above discussion moves
>>>> forward,
>>>>>> the
>>>>>>>>>> option 2
>>>>>>>>>>>>>>> solution
>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1
>>>>>>>>>>>>>>>>>>>>>> Yes, after discussing with Shammon
>>>>> offline,
>>>>>>> we
>>>>>>>>>> think
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> solution
>>>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>>>> be more suitable and also avoid any
>>>>>>>> inconsistency
>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> There are some inconsistent
>>>>> descriptions
>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>> content.
>>>>>>>>>>>>>>>> Would
>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>> to clean them up?
>>>>>>>>>>>>>>>>>>>>>> I will do my best to improve the
>>>> document
>>>>>> and
>>>>>>>>>>>> appreciate
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> @Shammon
>>>>>>>>>>>>>>>>>>>>>>> can you put the unselected option
>>> in
>>>>>>>> `Rejected
>>>>>>>>>>>>>>> Alternatives`
>>>>>>>>>>>>>>>>>>>>>> Sure, I have moved it to the
>>> `Rejected
>>>>>>>>>> Alternatives`.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 8:52 AM
>>> Shammon
>>>>> FY
>>>>>> <
>>>>>>>>>>>>>>> zjureel@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for your update.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I found there are two options in
>>>>>> `Proposed
>>>>>>>>>>> Changes`,
>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> put
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> unselected option in `Rejected
>>>>>>>> Alternatives`? I
>>>>>>>>>>> think
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>> help
>>>>>>>>>>>>>>>>>>>> us
>>>>>>>>>>>>>>>>>>>>>>> better understand your proposal
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 13, 2023 at 4:49 AM
>>> Jing
>>>> Ge
>>>>>>>>>>>>>>>>>> <jing@ververica.com.invalid
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for raising this FLIP. I
>>> am
>>>>>> still
>>>>>>>>>> confused
>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>> completely
>>>>>>>>>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>>>>>>>> the thread with following
>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 1. Naming confusion -
>>>>> registerCatalog()
>>>>>>> and
>>>>>>>>>>>>>> addCatalog()
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>> big
>>>>>>>>>>>>>>>>>>>>>>>> difference based on their names.
>>>> One
>>>>> of
>>>>>>>> them is
>>>>>>>>>>>>>>> responsible
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>>>> persistence. How about
>>>>>> persistCatalog()?
>>>>>>>>>>>>>>>>>>>>>>>> 2. As you mentioned that
>>>> Map<String,
>>>>>>>> Catalog>
>>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>>>>>>> and catalogStore is used for
>>> data
>>>>>>>> persistence.
>>>>>>>>>> I
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>> describing
>>>>>>>>>>>>>>>>>>>>>>>> their purpose conceptually and
>>>>> clearly
>>>>>> in
>>>>>>>> the
>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>> Some
>>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>>>>>>> features should be implemented,
>>>> i.e.
>>>>>> data
>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>> be consistent. Same Catalog
>>>> instance
>>>>>>>> should be
>>>>>>>>>>>> found
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the cache(either it has been
>>>>>> initialized
>>>>>>>> or it
>>>>>>>>>>> will
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> lazy
>>>>>>>>>>>>>>>>>>>>>> initialized)
>>>>>>>>>>>>>>>>>>>>>>>> for the same catalog name. The
>>>>>>> consistency
>>>>>>>> will
>>>>>>>>>>> be
>>>>>>>>>>>>>> taken
>>>>>>>>>>>>>>>> care
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> while
>>>>>>>>>>>>>>>>>>>>>>>> updating the catalog.
>>>>>>>>>>>>>>>>>>>>>>>> 3. As the above discussion moves
>>>>>> forward,
>>>>>>>> the
>>>>>>>>>>>> option
>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>> solution
>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>> like a replacement of option 1,
>>>>>> because,
>>>>>>>> afaiu,
>>>>>>>>>>>>> issues
>>>>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>>>> previously with option 1 are not
>>>>> solved
>>>>>>>> yet. Do
>>>>>>>>>>> you
>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> propose
>>>>>>>>>>>>>>>>>>>>>>>> both options and ask for
>>>> suggestions
>>>>>> for
>>>>>>>> both
>>>>>>>>>> of
>>>>>>>>>>>>> them?
>>>>>>>>>>>>>>>>>>>>>>>> 4. After you updated the FLIP,
>>>> there
>>>>>> are
>>>>>>>> some
>>>>>>>>>>>>>>> inconsistent
>>>>>>>>>>>>>>>>>>>>> descriptions
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the content. Would you like to
>>>> clean
>>>>>>> them
>>>>>>>> up?
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 7, 2023 at 9:24 AM
>>> Feng
>>>>>> Jin <
>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> hi Shammon
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response,
>>> and
>>>> I
>>>>>>>> completely
>>>>>>>>>>>> agree
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> view.
>>>>>>>>>>>>>>>>>>>>>>>>> Initially, I may have over
>>>>>> complicated
>>>>>>>> the
>>>>>>>>>>> whole
>>>>>>>>>>>>>> issue.
>>>>>>>>>>>>>>>>> First
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> foremost,
>>>>>>>>>>>>>>>>>>>>>>>>> we need to consider the
>>>> persistence
>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> Catalog's
>>>>>>>>>>>>>>>>>>>> Configuration.
>>>>>>>>>>>>>>>>>>>>>>>>> If we only need to provide
>>>>>> persistence
>>>>>>>> for
>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>> Configuration,
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> add a toConfiguration method
>>> to
>>>> the
>>>>>>>> Catalog
>>>>>>>>>>>>>> interface.
>>>>>>>>>>>>>>>>>>>>>>>>> This method can convert a
>>> Catalog
>>>>>>>> instance
>>>>>>>>>> to a
>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>> String>
>>>>>>>>>>>>>>>>>>>>>>>>> properties, and the default
>>>>>>>> implementation
>>>>>>>>>> will
>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> public interface Catalog {
>>>>>>>>>>>>>>>>>>>>>>>>> /**
>>>>>>>>>>>>>>>>>>>>>>>>> * Returns a map containing
>>> the
>>>>>>>> properties
>>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>> object.
>>>>>>>>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>>>>>>>> * @return a map containing
>>> the
>>>>>>>> properties
>>>>>>>>>> of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>> object
>>>>>>>>>>>>>>>>>>>>>>>>> * @throws
>>>>>>>> UnsupportedOperationException if
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> implementing
>>>>>>>>>>>>>>>>>>>>> class
>>>>>>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>>>>>>>> not override
>>>>>>>>>>>>>>>>>>>>>>>>> * the default
>>> implementation
>>>> of
>>>>>> this
>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> */
>>>>>>>>>>>>>>>>>>>>>>>>> default Map<String, String>
>>>>>>>> toProperties()
>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>>> throw new
>>>>>>>>>>>> UnsupportedOperationException("Please
>>>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>>>>>> toProperties
>>>>>>>>>>>>>>>>>>>>>>>>> for this catalog");
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The specific process is as
>>>> follows:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 1. If the user has
>>> configured a
>>>>>>>>>> CatalogStore,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> toConfiguration()
>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> will be called when
>>> registering a
>>>>>>> Catalog
>>>>>>>>>>>> instance
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>>>> catalogName,
>>>>>>>> Catalog
>>>>>>>>>>>>> catalog).
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>>>>> will be converted to a
>>>> Map<String,
>>>>>>>> String>
>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> saved
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore. If some Catalog
>>>>>> instances
>>>>>>>> do
>>>>>>>>>> not
>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> exception will be thrown.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2. If the user does not use a
>>>>>>>> CatalogStore,
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> toConfiguration()
>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> will not be called, ensuring
>>>>>>> consistency
>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 3. Saving both the Map<String,
>>>>>> Catalog>
>>>>>>>>>>> catalogs
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>> the same time can also avoid
>>>>>> conflicts
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> For lazy initialization:
>>>>>>>>>>>>>>>>>>>>>>>>> we can start from the Catalog
>>>>> itself
>>>>>>> and
>>>>>>>>>>> provide
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> dedicated
>>>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>>> implementation for lazy
>>> loading,
>>>>> such
>>>>>>> as
>>>>>>>>>>>>>>>>> ConfigurationCatalog
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> public class
>>> ConfigurationCatalog
>>>>>>>> implements
>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>>>
>>> ConfigurationCatalog(Map<String,
>>>>>>> String>
>>>>>>>>>>>>> properties)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I have added this design to
>>> the
>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> Feng
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 6, 2023 at
>>> 10:03 AM
>>>>>> Shammon
>>>>>>>> FY <
>>>>>>>>>>>>>>>>>> zjureel@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your answer.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I have no questions about
>>> the
>>>>>>>> functionality
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> `CatalogStore`,
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> different operations of
>>>> multiple
>>>>>>>>>>>>> `registerCatalog`
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`. The
>>>>>> implementation
>>>>>>> in
>>>>>>>>>> Trino
>>>>>>>>>>>> is
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> same,
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` in trino
>>> only
>>>>> has
>>>>>>> one
>>>>>>>>>>>>>>> `createCatalog`,
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to memory and
>>>>>> `CatalogStore`.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I think we don't need to add
>>>>>> another
>>>>>>>>>>>>>>> `registerCatalog`
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> `addCatalog`
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> save a catalog in
>>> `Map<String,
>>>>>>> Catalog>
>>>>>>>>>>>> catalogs`
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> `CatalogStore
>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore`. As you
>>>> mentioned
>>>>>>>> before,
>>>>>>>>>> the
>>>>>>>>>>>>>>>> `Map<String,
>>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs` is a cache in
>>>>>>>> `CatalogManager`.
>>>>>>>>>> How
>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>> saving
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> `Map<String, Catalog>
>>> catalogs`
>>>>> and
>>>>>>>>>>>> `CatalogStore
>>>>>>>>>>>>>>>>>>> catalogStore`
>>>>>>>>>>>>>>>>>>>>>>>> together
>>>>>>>>>>>>>>>>>>>>>>>>>> when it is registered or
>>> added
>>>> in
>>>>>>>>>>>>> `CatalogManager`?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>>> 5:22 PM
>>>>> Feng
>>>>>>>> Jin <
>>>>>>>>>>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your reply.
>>> I
>>>> am
>>>>>> very
>>>>>>>> sorry
>>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> misunderstanding
>>>>>>>>>>>>>>>>>>>>>>>>> caused
>>>>>>>>>>>>>>>>>>>>>>>>>>> by my deviation from the
>>>>> original
>>>>>>>>>>> discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>>>>>> pre-discussion
>>>>>>>> [1]
>>>>>>>>>> for
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, there was indeed
>>> such a
>>>>>>>> discussion
>>>>>>>>>>>> before.
>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> designing
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> whole solution, I found
>>> that
>>>>> the
>>>>>>>> logic of
>>>>>>>>>>>>>>>>> CatalogManager
>>>>>>>>>>>>>>>>>>>> itself
>>>>>>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>>>>>>> need to change much. *We
>>>> cannot
>>>>>>> only
>>>>>>>>>>> persist
>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>>>>>>>>>>>>>> themselves*,
>>>>>>>>>>>>>>>>>>>>>>>>>>> so exposing only
>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>>>>>>>>> not be enough to save
>>>> Catalogs,
>>>>>>>> because
>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> save
>>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations
>>>>> corresponding
>>>>>> to
>>>>>>>> the
>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>> instances.
>>>>>>>>>>>>>>>>>>>>>>>> Therefore,
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>> decided to introduce the
>>>>>>> CatalogStore
>>>>>>>>>>>> interface
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>>>>>>> persistence. Regarding
>>> this
>>>>>> part, I
>>>>>>>> also
>>>>>>>>>>> took
>>>>>>>>>>>>>>>>> inspiration
>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>> Trino's
>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation[1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> @yuxia
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>>>> handling
>>>>>>>> catalog
>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>> looks a
>>>>>>>>>>>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>>>> suggestion. I
>>>>>>> will
>>>>>>>>>>> provide
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> detailed
>>>>>>>>>>>>>>>>>>>>>> description
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> code examples for this
>>> part,
>>>>> and
>>>>>>> add
>>>>>>>> it
>>>>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method name
>>>>>>>> `addCatalog`
>>>>>>>>>>> still
>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>> confused
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for storing
>>>>>> catalog
>>>>>>> to
>>>>>>>>>>>>>> CatalogStore,
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`? It's
>>> very
>>>>>>> personal
>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> decide
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>>>>>>>>>>>>>>>>>>>>>>>>>>> StoreCatalog looks more
>>>>> intuitive
>>>>>>> to
>>>>>>>> me.
>>>>>>>>>> I
>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.For CREATE CATALOG
>>>>> statement,
>>>>>>>> which
>>>>>>>>>>>> method
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> called?
>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>>>>>> `addCatalog`?
>>>>>>>> I'm
>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with SQL
>>>>>> stement.
>>>>>>>>>>>>>>>>>>>>>>>>>>> For CREATE CATALOG, my
>>>> original
>>>>>>>> design
>>>>>>>>>> was
>>>>>>>>>>> to
>>>>>>>>>>>>> add
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> directly
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore, but this
>>> would
>>>>>>> disrupt
>>>>>>>> the
>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>> logic.
>>>>>>>>>>>>>>>>>>>>>>> Therefore, I
>>>>>>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>>>>>> we can do both: save the
>>>>>>>> configuration to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> initialize
>>>>>>>>>>>>>>>>>>>>>>>>>>> the Catalog instance at
>>> the
>>>>> same
>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Is it really
>>> neccessary
>>>> to
>>>>>>>> provide a
>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>> implmentation
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>>>>> `CatalogStoreFactory`?
>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is necessary,
>>>>>> otherwise
>>>>>>> we
>>>>>>>>>> would
>>>>>>>>>>>>> need
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> additional
>>>>>>>>>>>>>>>>>>>>>>>>>>> Map to store the
>>>> configuration
>>>>>> for
>>>>>>>> lazy
>>>>>>>>>>>>> loading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>>>>>>> registration
>>>>>>>> for
>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think
>>>>>>> registerCatalog(String
>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> made into an asynchronous
>>>>>> interface
>>>>>>>>>> because
>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> instance.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like lazy
>>>>>>>> initialization
>>>>>>>>>>> for
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Indeed, my description was
>>>>>>>> inaccurate. It
>>>>>>>>>>>>> should
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> lazy
>>>>>>>>>>>>>>>>>>>>>>> registration
>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of asynchronous
>>>>>>>> registration. I
>>>>>>>>>>> have
>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> title
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> the FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>>>> 4:27 PM
>>>>>>>> Shammon
>>>>>>>>>> FY <
>>>>>>>>>>>>>>>>>>> zjureel@gmail.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think if there is a
>>>>>>>> `registerCatalog`
>>>>>>>>>>>>> method
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager`,
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse users whether a
>>>>> method
>>>>>>>> named
>>>>>>>>>>>>>> `addCatalog`
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> `storeCatalog`
>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> added.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> And as you mentioned,
>>> the
>>>>>> memory
>>>>>>>>>> catalog
>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>> `cache`,
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> concept
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of `cache` should not be
>>>>>> exposed
>>>>>>> to
>>>>>>>>>>> users.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I found there is a
>>>>>> pre-discussion
>>>>>>>> [1]
>>>>>>>>>> for
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>> Please
>>>>>>>>>>>>>>>>>>>>>>> correct
>>>>>>>>>>>>>>>>>>>>>>>>> me
>>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm wrong, IIUC, the
>>>>> conclusion
>>>>>>> of
>>>>>>>> that
>>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> `CatalogManager` as an
>>>>>> interface
>>>>>>>> and
>>>>>>>>>>>>> implement
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>> stores
>>>>>>>>>>>>>>>>>>>>>>>>>>> such
>>>>>>>>>>>>>>>>>>>>>>>>>>>> as memory, file and
>>>> external
>>>>>>>> system.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think there is a gap
>>>>> between
>>>>>>> the
>>>>>>>>>>> current
>>>>>>>>>>>>> FLIP
>>>>>>>>>>>>>>>>> design
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> conclusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> What about the proposal
>>> of
>>>>> the
>>>>>>>>>> discussion
>>>>>>>>>>>> in
>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>>>> [1] ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Shammon FY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 4, 2023 at
>>>>> 3:41 PM
>>>>>>>> yuxia <
>>>>>>>>>>>>>>>>>>>>>>> luoyuxia@alumni.sjtu.edu.cn>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Feng for
>>> driving
>>>>> this
>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have few comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1: The mechanism of
>>>>> handling
>>>>>>>> catalog
>>>>>>>>>>> with
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>> looks a
>>>>>>>>>>>>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> complex to me. I think
>>>>> it'll
>>>>>> be
>>>>>>>>>> better
>>>>>>>>>>> to
>>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> java
>>>>>>>>>>>>>>>>>>>>>>>>> doc
>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these methods and
>>> give a
>>>>>> brief
>>>>>>>>>> example
>>>>>>>>>>> in
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> FLIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2: TBH, the method
>>> name
>>>>>>>> `addCatalog`
>>>>>>>>>>>> still
>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>> confused
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this method is for
>>>> storing
>>>>>>>> catalog to
>>>>>>>>>>>>>>>> CatalogStore,
>>>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to `storeCatalog`?
>>> It's
>>>>> very
>>>>>>>> personal
>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> decide
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not by your self.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: For CREATE CATALOG
>>>>>>> statement,
>>>>>>>>>> which
>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> called?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `registerCatalog` or
>>>>>>>> `addCatalog`?
>>>>>>>>>> I'm
>>>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>> add a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog to store with
>>> SQL
>>>>>>>> stement.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3: Is it really
>>>> neccessary
>>>>> to
>>>>>>>>>> provide a
>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>> implmentation
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface
>>>>>>> `CatalogStoreFactory`?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4: About asynchronous
>>>>>>>> registration
>>>>>>>>>> for
>>>>>>>>>>>>>> catalog.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When creating a
>>> catalog
>>>>>> with
>>>>>>>> CREATE
>>>>>>>>>>>>>> CATALOG,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>>>> used
>>>>>> by
>>>>>>>>>> default.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If asynchronous
>>>>> registration
>>>>>> is
>>>>>>>> the
>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>> behavior,
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that user can switch
>>> to
>>>>>>>> synchronous
>>>>>>>>>>>>>>> registration
>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>> before?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Will both method
>>>>> `addCatalog`
>>>>>>> and
>>>>>>>>>>>>>>>> `registerCatalog`
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IIUC, in asynchronous
>>>>>>>> registration,
>>>>>>>>>> it
>>>>>>>>>>>> may
>>>>>>>>>>>>>> well
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> CREATE
>>>>>>>>>>>>>>>>>>>>>>>> CATALOG
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> executes successfully,
>>>> but
>>>>>> then
>>>>>>>> the
>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>> CREATE
>>>>>>>>>>>>>>>>>>>> TABLE
>>>>>>>>>>>>>>>>>>>>>>>>> statement
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail for the catalog
>>> fail
>>>>> to
>>>>>>>> open.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's a break
>>>> change
>>>>>>> which
>>>>>>>>>>> should
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> highlighted
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> FLIP,
>>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be in compatibility
>>> part.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, by saying
>>>> asynchronous
>>>>>>>>>>>> registration, I
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> expect
>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be an executor to
>>>> open
>>>>>> or
>>>>>>>>>> register
>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> background,
>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> your previous
>>> comments,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "the Catalog instance
>>>> will
>>>>> be
>>>>>>>>>>> initialized
>>>>>>>>>>>>> if
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>>> initialized
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet. If the
>>>> initialization
>>>>>>>> process
>>>>>>>>>>> fails,
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>> statements
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> executed
>>> successfully."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks more like
>>> lazy
>>>>>>>>>> initialization
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yuxia
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ----- 原始邮件 -----
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发件人: "Feng Jin" <
>>>>>>>>>> jinfeng1995@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 收件人: "dev" <
>>>>>>> dev@flink.apache.org
>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 发送时间: 星期一, 2023年 4 月
>>> 03日
>>>> 下午
>>>>>>>> 3:27:45
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 主题: Re: [DISCUSS] FLIP
>>>> 295:
>>>>>>>> Support
>>>>>>>>>>>>>> persistence
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and asynchronous
>>>>> registration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone, Thank you
>>>> all
>>>>>> for
>>>>>>>> your
>>>>>>>>>>>>> interest
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> DISCUSS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Shammon
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How to handle a
>>> catalog
>>>>>> with
>>>>>>>> the
>>>>>>>>>> same
>>>>>>>>>>>>> name
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> them?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe this is a
>>>> crucial
>>>>>>>> point.
>>>>>>>>>>> Based
>>>>>>>>>>>> on
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>> current
>>>>>>>>>>>>>>>>>>>>>> personal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding, the
>>>>>> Map<String,
>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> serve
>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> cache
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for instantiated
>>> catalogs
>>>>> and
>>>>>>>> have
>>>>>>>>>> the
>>>>>>>>>>>>>> highest
>>>>>>>>>>>>>>>>>>> priority.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are three
>>> methods
>>>>> that
>>>>>>> can
>>>>>>>>>> affect
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
>>> registerCatalog(String
>>>>>>>>>> catalogName,
>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method puts the
>>>>> catalog
>>>>>>>> instance
>>>>>>>>>>>> into
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>>>> unregisterCatalog(String
>>>>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>>> the
>>>>>> catalog
>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> catalogName
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the Map<String,
>>>>> Catalog>
>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. getCatalog(String
>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method first
>>>> retrieves
>>>>>> the
>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Map<String,
>>> Catalog>
>>>>>>>> catalogs. If
>>>>>>>>>>> the
>>>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> exist,
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> retrieves the
>>>> corresponding
>>>>>>>>>>> configuration
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> CatalogStore,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initializes it, and
>>> puts
>>>>> the
>>>>>>>>>>> initialized
>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following two
>>> methods
>>>>>> only
>>>>>>>> modify
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. addCatalog(String
>>>>>>> catalogName,
>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>> String>
>>>>>>>>>>>>>>>>>>>>>>> properties)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method saves the
>>>>>>> properties
>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> catalogStore
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether there is a
>>>>>> catalogName
>>>>>>>> with
>>>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
>>> removeCatalog(String
>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This method removes
>>> the
>>>>>>> specified
>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> specified
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName in the
>>>>>>> catalogStore.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The following are
>>>> possible
>>>>>>>> conflict
>>>>>>>>>>>>>> scenarios:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When the
>>> corresponding
>>>>>>>> catalogName
>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore but not
>>> in
>>>> the
>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>> Catalog>,
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>>>>>>>> catalogName,
>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> succeed and be
>>> directly
>>>>> saved
>>>>>>> to
>>>>>>>> the
>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. When the
>>> corresponding
>>>>>>>> catalogName
>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogStore and the
>>>>>>> Map<String,
>>>>>>>>>>>> Catalog>,
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName, Catalog
>>>>> catalog)
>>>>>>>> method
>>>>>>>>>>> will
>>>>>>>>>>>>>> fail.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. When the
>>> corresponding
>>>>>>>> catalogName
>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> exists
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>,
>>> the
>>>>>>>>>>>> addCatalog(String
>>>>>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties)
>>>> method
>>>>>> can
>>>>>>>>>> directly
>>>>>>>>>>>>> save
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogStore, but the
>>>>>>>>>> getCatalog(String
>>>>>>>>>>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the new properties
>>>> for
>>>>>>>>>>> initialization
>>>>>>>>>>>>>>> because
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog instance
>>> already
>>>>>> exists
>>>>>>>> in
>>>>>>>>>>>> catalogs
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> prioritized.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, using the
>>>>>>>>>>>>> unregisterCatalog(String
>>>>>>>>>>>>>>>>>>>> catalogName)
>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> remove the instance
>>>>>>>> corresponding to
>>>>>>>>>>> the
>>>>>>>>>>>>>>> original
>>>>>>>>>>>>>>>>>>>>> catalogName
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it will
>>> confuse
>>>>>> users
>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> `registerCatalog(String
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogName,Catalog
>>>>> catalog)`
>>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>>>> `Map<String,
>>>>>>>>>>>>>>>>>>>> Catalog>
>>>>>>>>>>>>>>>>>>>>>>>>> catalogs`
>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>> `registerCatalog(String
>>>>>>>> catalogName,
>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>> String>
>>>>>>>>>>>>>>>>>>>>>>>>> properties)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This could potentially
>>>> lead
>>>>>> to
>>>>>>>>>>>> confusion. I
>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method name, perhaps
>>> to
>>>>>>>>>>> addCatalog(String
>>>>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>>>>>>> Map<String,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> String> properties),
>>> as
>>>>>>>> previously
>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Hang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
>>>>> `registerCatalog(String
>>>>>>>>>>>>>>> catalogName,Catalog
>>>>>>>>>>>>>>>>>>>> catalog,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean lazyInit)`
>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since a catalog is
>>>> already
>>>>> an
>>>>>>>>>> instance,
>>>>>>>>>>>>>> adding
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> "lazyInit"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter to the
>>>>>>>>>> registerCatalog(String
>>>>>>>>>>>>>>>>> catalogName,
>>>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>>> catalog)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method may not
>>>> necessarily
>>>>>>>> result in
>>>>>>>>>>> lazy
>>>>>>>>>>>>>>>>>>> initialization.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we need to think
>>>> about
>>>>>>>>>> encryption
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think encryption is
>>>>>>> necessary,
>>>>>>>> but
>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> encryption
>>>>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be implemented
>>> in
>>>>> the
>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> saved. For instance,
>>> the
>>>>>>>>>>> FileCatalogStore
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> encrypt
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is saved to a file,
>>>>> while
>>>>>>> the
>>>>>>>>>>>>>>>> MemoryCatalogStore
>>>>>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encryption.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need
>>> the
>>>>>>>>>>>>>> `MemoryCatalogStore`?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it is
>>> necessary
>>>> to
>>>>>>> have a
>>>>>>>>>>>>>>>>> MemoryCatalogStore
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> default
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation, which
>>>> saves
>>>>>>>> Catalog
>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, if we want
>>> to
>>>>>>>> implement
>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>> loading
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> Catalog,
>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would need to
>>> introduce
>>>>>>>> additional
>>>>>>>>>>> cache.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Xianxun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What if asynchronous
>>>>>>>> registration
>>>>>>>>>>>> failed?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is also a
>>> critical
>>>>>>> concern.
>>>>>>>>>> When
>>>>>>>>>>>>>>> executing
>>>>>>>>>>>>>>>>> DDL,
>>>>>>>>>>>>>>>>>>>> DQL,
>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> DML
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statements that
>>>> reference a
>>>>>>>> specific
>>>>>>>>>>>>> Catalog,
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be initialized
>>> if it
>>>>> has
>>>>>>> not
>>>>>>>>>> been
>>>>>>>>>>>>>>>> initialized
>>>>>>>>>>>>>>>>>> yet.
>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initialization process
>>>>> fails,
>>>>>>>> these
>>>>>>>>>>>>>> statements
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> executed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nt. The Map<String,
>>>>>> Catalog>
>>>>>>>>>> catalogs
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>>>>>>> catalogstore
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the CatalogManager all
>>>> have
>>>>>> the
>>>>>>>> same
>>>>>>>>>>>>> catalog
>>>>>>>>>>>>>>>> name,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>> correponding
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>>>>> instances.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This issue can be
>>>> resolved
>>>>> by
>>>>>>>>>> referring
>>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>>>>>>> responses.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> key principle is that
>>> the
>>>>>>> Catalog
>>>>>>>>>> that
>>>>>>>>>>>> has
>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>> recently
>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be given priority
>>>> for
>>>>>>>> subsequent
>>>>>>>>>>>> use.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 2, 2023 at
>>>>>> 10:58 PM
>>>>>>>>>> Xianxun
>>>>>>>>>>>> Ye
>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> yesorno828423@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Feng,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving
>>> this
>>>>>> Flip,
>>>>>>>> I do
>>>>>>>>>>>>> believe
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> Flip
>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> helpful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for users. The firm I
>>>> work
>>>>>> for
>>>>>>>> also
>>>>>>>>>>>>> manages a
>>>>>>>>>>>>>>> lot
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> catalogs,
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> submission of tasks
>>>> becomes
>>>>>>> slow
>>>>>>>>>>> because
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> loading a
>>>>>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We obtain the
>>> catalogs in
>>>>> the
>>>>>>>> user's
>>>>>>>>>>>> Flink
>>>>>>>>>>>>>> SQL
>>>>>>>>>>>>>>>>>> through
>>>>>>>>>>>>>>>>>>>>>> regular
>>>>>>>>>>>>>>>>>>>>>>>>>>> expression
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid loading all
>>>>> catalogs
>>>>>>> to
>>>>>>>>>>> improve
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> speed
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> task
>>>>>>>>>>>>>>>>>>>>>>>>>> submission.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After reading
>>> flip-295, I
>>>>>> have
>>>>>>>> some
>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. When creating a
>>>>> catalog
>>>>>>> with
>>>>>>>>>>> CREATE
>>>>>>>>>>>>>>> CATALOG,
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> registration method is
>>>> used
>>>>>> by
>>>>>>>>>> default.
>>>>>>>>>>>>> What
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>>>>>>>>>>> registration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed? And how to
>>>>> implement
>>>>>>>>>>> asynchronous
>>>>>>>>>>>>>>>>>> registration?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I also have the
>>> same
>>>>>>>> question
>>>>>>>>>> with
>>>>>>>>>>>>>>> Shammon’s
>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>>>>> comment.
>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Map<String, Catalog>
>>>>> catalogs
>>>>>>> and
>>>>>>>>>>>>>> CatalogStore
>>>>>>>>>>>>>>>>>>>> catalogstore
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CatalogManager all
>>> have
>>>> the
>>>>>>> same
>>>>>>>>>>> catalog
>>>>>>>>>>>>>> name,
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> correponding
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different catalog
>>>>> instances.
>>>>>>> For
>>>>>>>>>>> example:
>>>>>>>>>>>>>>>>> catalog1->
>>>>>>>>>>>>>>>>>>>>>>> jdbcCatalog
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalogs, but
>>> catalog1 ->
>>>>>>>> hiveCatalog
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> catalogstore.
>>>>>>>>>>>>>>>>>>>>>>> Would
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen? Maybe we
>>> should
>>>>>> define
>>>>>>>>>>> `catalogs`
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> `catalogstore`
>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clearly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Xianxun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年3月31日
>>> 11:13，Hang
>>>>>> Ruan
>>>>>>> <
>>>>>>>>>>>>>>>>>> ruanhang1993@gmail.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 写道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MemoryCatalogStore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi everyone,
Thank you everyone for your valuable input. If there are no further
questions or concerns about the FLIP[1], I would like to start voting
tomorrow (6/7).
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Best,
Feng
On Sun, Jun 4, 2023 at 3:33 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Samrat,
>
> Thanks for your advice.
>
> > 1. The createCatalog method does not mention any exceptions being
> thrown.
>
> CreateCatalog will throw CatalogException like registerCatalog. As
> CatalogException is a RuntimeException,
> there is no explicit declaration of throwing Exceptions in CatalogManager
> and TableEnvironment.
> To avoid misunderstandings, I have added the "throw CatalogException" flag
> to the createCatalog method definition of CatalogStore.
>
> > 2. Could you please provide an exhaustive list of the supported kinds?
>
> Sure, the documentation now includes the configuration of the built-in
> CatalogStore as well as how to configure a custom CatalogStore.
>
>
> Best,
> Feng
>
>
> On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com> wrote:
>
>> Hi Feng,
>>
>> Thank you for providing the proposal. I believe this feature will be
>> highly
>> valuable.
>>
>> I have a couple of inquiries:
>>
>> 1. According to the documentation [1], the createCatalog method does not
>> mention any exceptions being thrown. However, I would like to confirm if
>> it
>> is always true that there will be no failures in all scenarios. Please let
>> me know if there is any additional information I may have missed.
>>
>> 2. Regarding the registration process using the `table.catalog-store.kind`
>> configuration, could you please provide an exhaustive list of the
>> supported
>> kinds?
>> It would be great to have a comprehensive understanding of the options
>> available.
>>
>> Bests,
>> Samrat
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>>
>> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
>>
>> > Hi Hang and Jark
>> >
>> > Thank you very much for your reply.
>> >
>> > @Jark
>> > > 1. Could you move the CatalogStore registration API to the "Public
>> > Interface" section?
>> >
>> > Indeed, it is more reasonable.
>> >
>> > > 2. We should prefix "table." for the CatalogStore configuration.
>> >
>> > Sure, the names that you provided are indeed more logical and easier
>> to
>> > understand.
>> >
>> > > 3. About the open/close method in CatalogStoreFactory.
>> >
>> > I also agree with the proposed requirement scenario and design. I have
>> > already made modifications to the interface.
>> >
>> > @Hang
>> >
>> > > 1. The `CatalogStore` need the `open`/`close` methods to init or close
>> > the resource.
>> >
>> > I have already added the missing method.
>> >
>> > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> >
>> > Sorry for the misspelling, I updated this part.
>> >
>> > > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
>> > to init or close its resource
>> >
>> > This is indeed a scenario that needs to be considered, and I have added
>> > relevant methods accordingly.
>> > Additionally, I have included some relevant description information in
>> the
>> > documentation to help others understand
>> > why the open/close methods need to be added.
>> >
>> >
>> > Best,
>> > Feng
>> >
>> > On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
>> >
>> > > Hi Jing,
>> > >
>> > > Thank you for the update.
>> > >
>> > > 1. Could you move the CatalogStore registration API to the "Public
>> > > Interface" section?
>> > > "Proposed Changes" is more like a place to describe the implementation
>> > > details.
>> > >
>> > > 2. We should prefix "table." for the CatalogStore configuration.
>> Besides,
>> > > the config key
>> > > name should be hierarchical[1]. Therefore, it may be better to use:
>> > > "table.catalog-store.kind"
>> > > "table.catalog-store.file.path"
>> > >
>> > > 3. I think Hang's suggestions make sense.
>> > >
>> > > Others look good to me.
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com>
>> wrote:
>> > >
>> > > > Hi, Feng.
>> > > >
>> > > > Thanks for the update.
>> > > > The current design looks good to me. I have some minor comments.
>> > > >
>> > > > 1. The `CatalogStore` need the `open`/`close` methods to init or
>> close
>> > > the
>> > > > resource. For example, when storing the information in MySQL, the
>> store
>> > > > needs to open and close the connections.
>> > > >
>> > > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>> > > >
>> > > > 3. About the usage in the sql gateway.
>> > > > Considering the usage in sql gateway, the sql gateway may create a
>> > > > CatalogStore for each session.
>> > > > If we are using the MySqlCatalogStore, there would be so many
>> > > connections.
>> > > > How can we reuse the connection among these sessions?
>> > > > I think sql gateway need to maintain a connection pool in
>> > > > the CatalogStoreFactory and each session get its own connection from
>> > the
>> > > > pool when it is created.
>> > > > Then the `CatalogStoreFactory` may need the `open`/`close` methods
>> to
>> > > init
>> > > > or close its resource.
>> > > > Or is there a better way?
>> > > >
>> > > > Best,
>> > > > Hang
>> > > >
>> > > > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>> > > >
>> > > > > Thanks Jingsong.
>> > > > >
>> > > > > > Just naming, maybe `createCatalog` in TableEnv
>> > > > >
>> > > > > +1 For this, I have already updated FLIP.
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Feng
>> > > > >
>> > > > >
>> > > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
>> jingsonglee0@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Thanks Feng,
>> > > > > >
>> > > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
>> > > > > > functions are converted to createxxx from registerxxx.
>> > > > > >
>> > > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <jinfeng1995@gmail.com
>> >
>> > > > wrote:
>> > > > > > >
>> > > > > > > Hi jark, thanks for your suggestion.
>> > > > > > >
>> > > > > > > > 1. How to register the CatalogStore for Table API? I think
>> the
>> > > > > > > CatalogStore should be immutable once TableEnv is created.
>> > > Otherwise,
>> > > > > > there
>> > > > > > > might be some data inconsistencies when CatalogStore is
>> changed.
>> > > > > > >
>> > > > > > > Yes, We should initialize the CatalogStore when creating the
>> > > > TableEnv.
>> > > > > > > Therefore, my current proposal is to add a method to configure
>> > the
>> > > > > > > CatalogStore in EnvironmentSettings.
>> > > > > > >
>> > > > > > > // Initialize a catalog Store
>> > > > > > > CatalogStore catalogStore = new FileCatalogStore("");
>> > > > > > >
>> > > > > > > // set up the Table API
>> > > > > > > final EnvironmentSettings settings =
>> > > > > > >
>> EnvironmentSettings.newInstance().inBatchMode()
>> > > > > > > .withCatalogStore(catalogStore)
>> > > > > > > .build();
>> > > > > > >
>> > > > > > > final TableEnvironment tableEnv =
>> > > > > > TableEnvironment.create(settings);
>> > > > > > >
>> > > > > > >
>> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
>> > default
>> > > > > > method,
>> > > > > > > not an interface method?
>> > > > > > >
>> > > > > > > Sorry, While I did refer to the Catalog interface, I agree
>> that
>> > as
>> > > a
>> > > > > new
>> > > > > > > interface, the CatalogStoreFactory should not have a default
>> > method
>> > > > but
>> > > > > > an
>> > > > > > > interface method. I have already modified the interface.
>> > > > > > >
>> > > > > > > > 3. Please mention the alternative API in Javadoc for the
>> > > > > > > deprecated`registerCatalog`.
>> > > > > > > > 4. In the "Compatibility" section, would be better to
>> mention
>> > the
>> > > > > > changed
>> > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
>> > other
>> > > > > > > persisted catalog store) is used.
>> > > > > > >
>> > > > > > > Thanks for the suggestion, I have updated the FLIP [1].
>> > > > > > >
>> > > > > > >
>> > > > > > > [1].
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > > > > >
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Feng
>> > > > > > >
>> > > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
>> wrote:
>> > > > > > >
>> > > > > > > > Hi Feng,
>> > > > > > > >
>> > > > > > > > This is a useful FLIP. Thanks for starting this discussion.
>> > > > > > > > The current design looks pretty good to me. I just have some
>> > > minor
>> > > > > > > > comments.
>> > > > > > > >
>> > > > > > > > 1. How to register the CatalogStore for Table API? I think
>> the
>> > > > > > CatalogStore
>> > > > > > > > should be immutable once TableEnv is created. Otherwise,
>> there
>> > > > might
>> > > > > be
>> > > > > > > > some data inconsistencies when CatalogStore is changed.
>> > > > > > > >
>> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
>> > default
>> > > > > > method,
>> > > > > > > > not an interface method?
>> > > > > > > >
>> > > > > > > > 3. Please mention the alternative API in Javadoc for the
>> > > deprecated
>> > > > > > > > `registerCatalog`.
>> > > > > > > >
>> > > > > > > > 4. In the "Compatibility" section, would be better to
>> mention
>> > the
>> > > > > > changed
>> > > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
>> > > other
>> > > > > > > > persisted catalog store) is used.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Jark
>> > > > > > > >
>> > > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <
>> jinfeng1995@gmail.com>
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi ， thanks all for reviewing the flip.
>> > > > > > > > >
>> > > > > > > > > @Ron
>> > > > > > > > >
>> > > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore
>> > method,
>> > > > do
>> > > > > we
>> > > > > > > > need
>> > > > > > > > > to provide a default implementation?
>> > > > > > > > >
>> > > > > > > > > Yes, we will provide a default
>> InMemoryCatalogStoreFactory to
>> > > > > create
>> > > > > > an
>> > > > > > > > > InMemoryCatalogStore.
>> > > > > > > > >
>> > > > > > > > > > If we get a Catalog from CatalogStore, after
>> initializing
>> > > it,
>> > > > > > whether
>> > > > > > > > we
>> > > > > > > > > put it in Map<String, Catalog> catalogs again?
>> > > > > > > > >
>> > > > > > > > > Yes, in the current design, catalogs are stored as
>> snapshots,
>> > > and
>> > > > > > once
>> > > > > > > > > initialized, the Catalog will be placed in the Map<String,
>> > > > Catalog>
>> > > > > > > > > catalogs.
>> > > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be
>> the
>> > > > primary
>> > > > > > > > source
>> > > > > > > > > for obtaining the corresponding Catalog.
>> > > > > > > > >
>> > > > > > > > > > how about renaming them to `catalog.store.type` and
>> > > > > > > > > `catalog.store.path`?
>> > > > > > > > >
>> > > > > > > > > I think it is okay. Adding "sql" at the beginning may
>> seem a
>> > > bit
>> > > > > > > > strange. I
>> > > > > > > > > will update the FLIP.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > @Shammon
>> > > > > > > > >
>> > > > > > > > > Thank you for the review. I have made the necessary
>> > > corrections.
>> > > > > > > > > Regarding the modifications made to the Public Interface,
>> I
>> > > have
>> > > > > also
>> > > > > > > > > included the relevant changes to the `TableEnvironment`.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Feng
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <
>> > zjureel@gmail.com>
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi feng,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for updating, I have some minor comments
>> > > > > > > > > >
>> > > > > > > > > > 1. The modification of `CatalogManager` should not be in
>> > > > `Public
>> > > > > > > > > > Interfaces`, it is not a public interface.
>> > > > > > > > > >
>> > > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore`
>> and
>> > > > > > > > > > `CatalogStoreFactory`
>> > > > > > > > > >
>> > > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
>> > > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
>> > > should
>> > > > be
>> > > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
>> > > > > > > > > > catalogStore.get(catalogName);`
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Shammon FY
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <
>> > ron9.liu@gmail.com>
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi, Feng
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for driving this FLIP, this proposal is very
>> > useful
>> > > > for
>> > > > > > > > catalog
>> > > > > > > > > > > management.
>> > > > > > > > > > > I have some small questions:
>> > > > > > > > > > >
>> > > > > > > > > > > 1. Regarding the
>> CatalogStoreFactory#createCatalogStore
>> > > > method,
>> > > > > > do we
>> > > > > > > > > > need
>> > > > > > > > > > > to provide a default implementation?
>> > > > > > > > > > > 2. If we get Catalog from CatalogStore, after
>> > initializing
>> > > > it,
>> > > > > > > > whether
>> > > > > > > > > we
>> > > > > > > > > > > put it to Map<String, Catalog> catalogs again?
>> > > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
>> > > > > > > > > > > `sql.catalog.store.file.path`, how about renaming
>> them to
>> > > > > > > > > > > `catalog.store.type` and `catalog.store.path`?
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Ron
>> > > > > > > > > > >
>> > > > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一
>> 21:19写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi yuxia
>> > > > > > > > > > > >
>> > > > > > > > > > > > > But from the code in Proposed Changes, once we
>> > > register
>> > > > > the
>> > > > > > > > > Catalog,
>> > > > > > > > > > > we
>> > > > > > > > > > > > initialize it and open it. right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
>> > > > original
>> > > > > > > > CREATE
>> > > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
>> > > > > > > > registerCatalog
>> > > > > > > > > so
>> > > > > > > > > > > > that parameter validation can be performed.
>> > > > > > > > > > > >
>> > > > > > > > > > > > In the current design, lazy initialization is mainly
>> > > > > reflected
>> > > > > > in
>> > > > > > > > > > > > getCatalog. If CatalogStore has already saved some
>> > > catalog
>> > > > > > > > > > > configurations,
>> > > > > > > > > > > > only initialization is required in getCatalog.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best,
>> > > > > > > > > > > > Feng
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
>> > > > > > luoyuxia@alumni.sjtu.edu.cn
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi, Feng.
>> > > > > > > > > > > > > I'm trying to understanding the meaning of *lazy
>> > > > > > initialization*.
>> > > > > > > > > If
>> > > > > > > > > > > i'm
>> > > > > > > > > > > > > wrong, please correct me.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > IIUC, lazy initialization means only you need to
>> > access
>> > > > the
>> > > > > > > > > catalog,
>> > > > > > > > > > > then
>> > > > > > > > > > > > > you initialize it. But from the code in Proposed
>> > > Changes,
>> > > > > > once we
>> > > > > > > > > > > > register
>> > > > > > > > > > > > > the Catalog,
>> > > > > > > > > > > > > we initialize it and open it. right?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > Yuxia
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > ----- 原始邮件 -----
>> > > > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
>> > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
>> > > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
>> > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
>> > > > Catalog
>> > > > > > > > > > > configuration
>> > > > > > > > > > > > > and asynchronous registration
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for your effort! +1 for the proposal.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > One of the major changes is that current design
>> will
>> > > > > provide
>> > > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead
>> > of a
>> > > > > > cache,
>> > > > > > > > > which
>> > > > > > > > > > > > means
>> > > > > > > > > > > > > once it has been initialized, any changes done by
>> > other
>> > > > > > sessions
>> > > > > > > > > will
>> > > > > > > > > > > not
>> > > > > > > > > > > > > affect it. Point 6 described follow-up options for
>> > > > further
>> > > > > > > > > > improvement.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > Jing
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
>> > > > > > jinfeng1995@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi all, I would like to update you on the latest
>> > > > progress
>> > > > > > of
>> > > > > > > > the
>> > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge,
>> Shammon
>> > FY,
>> > > > > > ShengKai
>> > > > > > > > > Fang
>> > > > > > > > > > > > and I
>> > > > > > > > > > > > > > had an offline discussion regarding the overall
>> > > > solution
>> > > > > > for
>> > > > > > > > > Flink
>> > > > > > > > > > > > > > CatalogStore. We have reached a consensus and I
>> > have
>> > > > > > updated
>> > > > > > > > the
>> > > > > > > > > > > final
>> > > > > > > > > > > > > > solution in FLIP.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Next, let me briefly describe the entire design:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 1.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
>> > > > > > configuration
>> > > > > > > > > > similar
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > TableDescriptor.
>> > > > > > > > > > > > > > 2.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The two key functions of CatalogStore - void
>> > > > > > > > > storeCatalog(String
>> > > > > > > > > > > > > > catalogName, CatalogDescriptor) and
>> > > > CatalogDescriptor
>> > > > > > > > > > > > > getCatalog(String)
>> > > > > > > > > > > > > > will both use CatalogDescriptor instead of
>> > Catalog
>> > > > > > instance.
>> > > > > > > > > > This
>> > > > > > > > > > > > way,
>> > > > > > > > > > > > > > CatalogStore will only be responsible for
>> saving
>> > > and
>> > > > > > > > > retrieving
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > configurations without having to initialize
>> > > > catalogs.
>> > > > > > > > > > > > > > 3.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The default registerCatalog(String
>> catalogName,
>> > > > > Catalog
>> > > > > > > > > catalog)
>> > > > > > > > > > > > > > function in CatalogManager will be marked as
>> > > > > deprecated.
>> > > > > > > > > > > > > > 4.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > A new function registerCatalog(String
>> > catalogName,
>> > > > > > > > > > > CatalogDescriptor
>> > > > > > > > > > > > > > catalog) will be added to serve as the
>> default
>> > > > > > registration
>> > > > > > > > > > > function
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > catalogs in CatalogManager.
>> > > > > > > > > > > > > > 5.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager
>> > will
>> > > > > > remain
>> > > > > > > > > > > unchanged
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > save initialized catalogs.This means that
>> > deletion
>> > > > > > > > operations
>> > > > > > > > > > from
>> > > > > > > > > > > > one
>> > > > > > > > > > > > > > session won't synchronize with other
>> sessions.
>> > > > > > > > > > > > > > 6.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > To support multi-session synchronization
>> > scenarios
>> > > > for
>> > > > > > > > > deletions
>> > > > > > > > > > > > later
>> > > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
>> > > > > > > > > configurable.There
>> > > > > > > > > > > may
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > three possible situations:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > a.Default caching of all initialized catalogs
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > b.Introduction of LRU cache logic which can
>> > > > > > automatically
>> > > > > > > > > clear
>> > > > > > > > > > > > > > long-unused catalogs.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > c.No caching of any instances; each call to
>> > > > getCatalog
>> > > > > > > > > creates a
>> > > > > > > > > > > new
>> > > > > > > > > > > > > > instance.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This is the document for discussion:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This is the final proposal document:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you very much for your attention and
>> > > suggestions
>> > > > on
>> > > > > > this
>> > > > > > > > > > > FLIP. A
>> > > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions
>> on
>> > > the
>> > > > > > entire
>> > > > > > > > > > design
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > organizing offline discussions.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > If you have any further suggestions or feedback
>> > about
>> > > > > this
>> > > > > > FLIP
>> > > > > > > > > > > please
>> > > > > > > > > > > > > feel
>> > > > > > > > > > > > > > free to share.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
>> > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good
>> to
>> > me.
>> > > > We
>> > > > > > could
>> > > > > > > > > > still
>> > > > > > > > > > > > > > > reconsider in the future how to provide more
>> > common
>> > > > > > built-in
>> > > > > > > > > > cache
>> > > > > > > > > > > > > > > functionality in CatalogManager so that not
>> every
>> > > > > > > > CatalogSotre
>> > > > > > > > > > > > > > > implementation has to take care of it.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
>> > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hi Jing,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Thanks for your reply.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > There might be more such issues. I would
>> > > suggest
>> > > > > you
>> > > > > > > > > > > completely
>> > > > > > > > > > > > > walk
>> > > > > > > > > > > > > > > > through the FLIP again and fix those issues
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I am very sorry for my carelessness and at
>> the
>> > > same
>> > > > > > time, I
>> > > > > > > > > > > greatly
>> > > > > > > > > > > > > > > > appreciate your careful review. I have
>> > thoroughly
>> > > > > > checked
>> > > > > > > > the
>> > > > > > > > > > > > entire
>> > > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > and made corrections to these issues.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > If I am not mistaken, with the current
>> FLIP
>> > > > > design,
>> > > > > > > > > > > > > CatalogManager
>> > > > > > > > > > > > > > > > could work without Optional CatalogStore
>> being
>> > > > > > configured.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Yes, in the original design, CatalogStore
>> was
>> > not
>> > > > > > necessary
>> > > > > > > > > > > because
>> > > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog>
>> > catalogs
>> > > > to
>> > > > > > store
>> > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > instances.
>> > > > > > > > > > > > > > > > However, this caused inconsistency issues.
>> > > > > Therefore, I
>> > > > > > > > > > modified
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > > > part
>> > > > > > > > > > > > > > > > of the design and removed Map<String,
>> Catalog>
>> > > > > catalogs
>> > > > > > > > from
>> > > > > > > > > > > > > > > > CatalogManager.
>> > > > > > > > > > > > > > > > At the same time, InMemoryCatalog will
>> serve
>> > as
>> > > > the
>> > > > > > > > default
>> > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > to save catalogs in memory and replace the
>> > > > > > functionality of
>> > > > > > > > > > > > > > > > Map<String,Catalog>catalogs.
>> > > > > > > > > > > > > > > > The previous plan that kept
>> > > > > > Map<String,Catalog>catalogs has
>> > > > > > > > > > been
>> > > > > > > > > > > > > moved
>> > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > Rejected Alternatives.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
>> > > > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > There are still many places contain
>> > > inconsistent
>> > > > > > content,
>> > > > > > > > > > e.g.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still
>> used.
>> > > > > > > > > > > > > > > > > 2. The java comment of the method
>> > > > > > registerCatalog(String
>> > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does
>> not
>> > > tell
>> > > > > > what the
>> > > > > > > > > > > method
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > doing.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > There might be more such issues. I would
>> > > suggest
>> > > > > you
>> > > > > > > > > > completely
>> > > > > > > > > > > > > walk
>> > > > > > > > > > > > > > > > > through the FLIP again and fix those
>> issues.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you
>> mean
>> > > that
>> > > > > you
>> > > > > > will
>> > > > > > > > > > build
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > functionality in the CatalogStore? This
>> is a
>> > > very
>> > > > > > > > different
>> > > > > > > > > > > > design
>> > > > > > > > > > > > > > > > concept
>> > > > > > > > > > > > > > > > > from what the current FLIP described. If
>> I am
>> > > not
>> > > > > > > > mistaken,
>> > > > > > > > > > > with
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > current FLIP design, CatalogManager could
>> > work
>> > > > > > without
>> > > > > > > > > > Optional
>> > > > > > > > > > > > > > > > > CatalogStore being configured. That is the
>> > > reason
>> > > > > > why I
>> > > > > > > > > > > mentioned
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > last email that the example code wrt the
>> > > Optional
>> > > > > > > > > > CatalogStore
>> > > > > > > > > > > is
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > correct.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
>> > > > > > > > > > > jinfeng1995@gmail.com>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Hi Jing
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > There are still some NIT issues in the
>> > FLIP
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Thank you very much for the careful
>> > review. I
>> > > > > have
>> > > > > > > > > already
>> > > > > > > > > > > made
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > relevant changes.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in
>> the
>> > > > > > > > multi-instance
>> > > > > > > > > > > > > > scenarios, I
>> > > > > > > > > > > > > > > > am
>> > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > sure if this is the intended behaviour
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Currently, there are conflicts in
>> multiple
>> > > > > > scenarios
>> > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > > > > > current
>> > > > > > > > > > > > > > > > > > design. I am thinking whether we should
>> > > remove
>> > > > > > > > > 'Map<String,
>> > > > > > > > > > > > > > Catalog>'
>> > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > make Cache the default behavior of
>> > > > > > > > InMemoryCatalogStore.
>> > > > > > > > > > This
>> > > > > > > > > > > > > way,
>> > > > > > > > > > > > > > > > users
>> > > > > > > > > > > > > > > > > > can implement their own CatalogStore to
>> > > achieve
>> > > > > > > > > > > multi-instance
>> > > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you
>> > think?
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
>> > > > > > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There
>> are
>> > > > still
>> > > > > > some
>> > > > > > > > > NIT
>> > > > > > > > > > > > issues
>> > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > > > > like:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore
>> > has
>> > > > been
>> > > > > > used
>> > > > > > > > as
>> > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > > instead of Optional in the code
>> example.
>> > It
>> > > > > > should be
>> > > > > > > > > > fine
>> > > > > > > > > > > to
>> > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > pseudo code for now and update it
>> after
>> > you
>> > > > > > submit
>> > > > > > > > the
>> > > > > > > > > > PR.
>> > > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used
>> > somewhere
>> > > in
>> > > > > the
>> > > > > > > > > > rejected
>> > > > > > > > > > > > > > section
>> > > > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > > should be persistContext(...) to keep
>> it
>> > > > > > consistent.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
>> > > > > > multi-instance
>> > > > > > > > > > > > > scenarios,
>> > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > am
>> > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > sure if this is the intended
>> behaviour.
>> > If
>> > > > > > > > Map<String,
>> > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > catalogs
>> > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > used as a cache, it should be invalid,
>> > once
>> > > > the
>> > > > > > > > related
>> > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > > been
>> > > > > > > > > > > > > > > > > > > removed from the CatalogStore by
>> another
>> > > > > > instance.
>> > > > > > > > Did
>> > > > > > > > > I
>> > > > > > > > > > > miss
>> > > > > > > > > > > > > > > > > something?
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng
>> Jin
>> > <
>> > > > > > > > > > > > > jinfeng1995@gmail.com>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Hi Jing,Shammon
>> > > > > > > > > > > > > > > > > > > > Thanks for your reply.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > @Jing
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > How about persistCatalog()?
>> > > > > > > > > > > > > > > > > > > > I think this is a good function
>> name, I
>> > > > have
>> > > > > > > > updated
>> > > > > > > > > it
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > documentation.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Some common cache features
>> should
>> > be
>> > > > > > > > implemented
>> > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
>> > > > alternative
>> > > > > 1
>> > > > > > > > turns
>> > > > > > > > > > out
>> > > > > > > > > > > to
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > appropriate later, I will improve
>> this
>> > > part
>> > > > > of
>> > > > > > the
>> > > > > > > > > > > content.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > As the above discussion moves
>> > forward,
>> > > > the
>> > > > > > > > option 2
>> > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > like a replacement of option 1
>> > > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
>> > > offline,
>> > > > > we
>> > > > > > > > think
>> > > > > > > > > > that
>> > > > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > 2
>> > > > > > > > > > > > > > > > > > > might
>> > > > > > > > > > > > > > > > > > > > be more suitable and also avoid any
>> > > > > > inconsistency
>> > > > > > > > > > issues.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > There are some inconsistent
>> > > descriptions
>> > > > in
>> > > > > > the
>> > > > > > > > > > > content.
>> > > > > > > > > > > > > > Would
>> > > > > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > > > > > > to clean them up?
>> > > > > > > > > > > > > > > > > > > > I will do my best to improve the
>> > document
>> > > > and
>> > > > > > > > > > appreciate
>> > > > > > > > > > > > your
>> > > > > > > > > > > > > > > > > > > suggestions.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > @Shammon
>> > > > > > > > > > > > > > > > > > > > > can you put the unselected option
>> in
>> > > > > > `Rejected
>> > > > > > > > > > > > > Alternatives`
>> > > > > > > > > > > > > > > > > > > > Sure, I have moved it to the
>> `Rejected
>> > > > > > > > Alternatives`.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Best
>> > > > > > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM
>> Shammon
>> > > FY
>> > > > <
>> > > > > > > > > > > > > zjureel@gmail.com>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Hi Feng
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Thanks for your update.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > I found there are two options in
>> > > > `Proposed
>> > > > > > > > > Changes`,
>> > > > > > > > > > > can
>> > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > put
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > unselected option in `Rejected
>> > > > > > Alternatives`? I
>> > > > > > > > > think
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > > may
>> > > > > > > > > > > > > > > > help
>> > > > > > > > > > > > > > > > > > us
>> > > > > > > > > > > > > > > > > > > > > better understand your proposal
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > Shammon FY
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM
>> Jing
>> > Ge
>> > > > > > > > > > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I
>> am
>> > > > still
>> > > > > > > > confused
>> > > > > > > > > > > after
>> > > > > > > > > > > > > > > > > completely
>> > > > > > > > > > > > > > > > > > > > > reading
>> > > > > > > > > > > > > > > > > > > > > > the thread with following
>> > questions:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
>> > > registerCatalog()
>> > > > > and
>> > > > > > > > > > > > addCatalog()
>> > > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > no
>> > > > > > > > > > > > > > > > > > big
>> > > > > > > > > > > > > > > > > > > > > > difference based on their names.
>> > One
>> > > of
>> > > > > > them is
>> > > > > > > > > > > > > responsible
>> > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > > > > > persistence. How about
>> > > > persistCatalog()?
>> > > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that
>> > Map<String,
>> > > > > > Catalog>
>> > > > > > > > > > > catalogs
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > > used
>> > > > > > > > > > > > > > > > > as a
>> > > > > > > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > > > > > > and catalogStore is used for
>> data
>> > > > > > persistence.
>> > > > > > > > I
>> > > > > > > > > > > would
>> > > > > > > > > > > > > > > suggest
>> > > > > > > > > > > > > > > > > > > > describing
>> > > > > > > > > > > > > > > > > > > > > > their purpose conceptually and
>> > > clearly
>> > > > in
>> > > > > > the
>> > > > > > > > > FLIP.
>> > > > > > > > > > > > Some
>> > > > > > > > > > > > > > > common
>> > > > > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > > > > > > features should be implemented,
>> > i.e.
>> > > > data
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > cache
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > store
>> > > > > > > > > > > > > > > > > > > > > should
>> > > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog
>> > instance
>> > > > > > should be
>> > > > > > > > > > found
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > store
>> > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > the cache(either it has been
>> > > > initialized
>> > > > > > or it
>> > > > > > > > > will
>> > > > > > > > > > > be
>> > > > > > > > > > > > > lazy
>> > > > > > > > > > > > > > > > > > > > initialized)
>> > > > > > > > > > > > > > > > > > > > > > for the same catalog name. The
>> > > > > consistency
>> > > > > > will
>> > > > > > > > > be
>> > > > > > > > > > > > taken
>> > > > > > > > > > > > > > care
>> > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > while
>> > > > > > > > > > > > > > > > > > > > > > updating the catalog.
>> > > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
>> > > > forward,
>> > > > > > the
>> > > > > > > > > > option
>> > > > > > > > > > > 2
>> > > > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
>> > > > because,
>> > > > > > afaiu,
>> > > > > > > > > > > issues
>> > > > > > > > > > > > > > > > mentioned
>> > > > > > > > > > > > > > > > > > > > > > previously with option 1 are not
>> > > solved
>> > > > > > yet. Do
>> > > > > > > > > you
>> > > > > > > > > > > > still
>> > > > > > > > > > > > > > > want
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > propose
>> > > > > > > > > > > > > > > > > > > > > > both options and ask for
>> > suggestions
>> > > > for
>> > > > > > both
>> > > > > > > > of
>> > > > > > > > > > > them?
>> > > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP,
>> > there
>> > > > are
>> > > > > > some
>> > > > > > > > > > > > > inconsistent
>> > > > > > > > > > > > > > > > > > > descriptions
>> > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > the content. Would you like to
>> > clean
>> > > > > them
>> > > > > > up?
>> > > > > > > > > > > Thanks!
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > Jing
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM
>> Feng
>> > > > Jin <
>> > > > > > > > > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > hi Shammon
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Thank you for your response,
>> and
>> > I
>> > > > > > completely
>> > > > > > > > > > agree
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > your
>> > > > > > > > > > > > > > > > > > point
>> > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > view.
>> > > > > > > > > > > > > > > > > > > > > > > Initially, I may have over
>> > > > complicated
>> > > > > > the
>> > > > > > > > > whole
>> > > > > > > > > > > > issue.
>> > > > > > > > > > > > > > > First
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > foremost,
>> > > > > > > > > > > > > > > > > > > > > > > we need to consider the
>> > persistence
>> > > > of
>> > > > > > the
>> > > > > > > > > > > Catalog's
>> > > > > > > > > > > > > > > > > > Configuration.
>> > > > > > > > > > > > > > > > > > > > > > > If we only need to provide
>> > > > persistence
>> > > > > > for
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > Configuration,
>> > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method
>> to
>> > the
>> > > > > > Catalog
>> > > > > > > > > > > > interface.
>> > > > > > > > > > > > > > > > > > > > > > > This method can convert a
>> Catalog
>> > > > > > instance
>> > > > > > > > to a
>> > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > String>
>> > > > > > > > > > > > > > > > > > > > > > > properties, and the default
>> > > > > > implementation
>> > > > > > > > will
>> > > > > > > > > > > throw
>> > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > exception.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > public interface Catalog {
>> > > > > > > > > > > > > > > > > > > > > > > /**
>> > > > > > > > > > > > > > > > > > > > > > > * Returns a map containing
>> the
>> > > > > > properties
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > object.
>> > > > > > > > > > > > > > > > > > > > > > > *
>> > > > > > > > > > > > > > > > > > > > > > > * @return a map containing
>> the
>> > > > > > properties
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > object
>> > > > > > > > > > > > > > > > > > > > > > > * @throws
>> > > > > > UnsupportedOperationException if
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > implementing
>> > > > > > > > > > > > > > > > > > > class
>> > > > > > > > > > > > > > > > > > > > > does
>> > > > > > > > > > > > > > > > > > > > > > > not override
>> > > > > > > > > > > > > > > > > > > > > > > * the default
>> implementation
>> > of
>> > > > this
>> > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > */
>> > > > > > > > > > > > > > > > > > > > > > > default Map<String, String>
>> > > > > > toProperties()
>> > > > > > > > {
>> > > > > > > > > > > > > > > > > > > > > > > throw new
>> > > > > > > > > > UnsupportedOperationException("Please
>> > > > > > > > > > > > > > > implement
>> > > > > > > > > > > > > > > > > > > > > > toProperties
>> > > > > > > > > > > > > > > > > > > > > > > for this catalog");
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > The specific process is as
>> > follows:
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 1. If the user has
>> configured a
>> > > > > > > > CatalogStore,
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > toConfiguration()
>> > > > > > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > will be called when
>> registering a
>> > > > > Catalog
>> > > > > > > > > > instance
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
>> > catalogName,
>> > > > > > Catalog
>> > > > > > > > > > > catalog).
>> > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > > > > will be converted to a
>> > Map<String,
>> > > > > > String>
>> > > > > > > > > > > properties
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > saved
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
>> > > > instances
>> > > > > > do
>> > > > > > > > not
>> > > > > > > > > > > > > implement
>> > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > method,
>> > > > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > > exception will be thrown.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
>> > > > > > CatalogStore,
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > > > toConfiguration()
>> > > > > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
>> > > > > consistency
>> > > > > > with
>> > > > > > > > > the
>> > > > > > > > > > > > > > original
>> > > > > > > > > > > > > > > > > > process.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
>> > > > Catalog>
>> > > > > > > > > catalogs
>> > > > > > > > > > > and
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > > > > > > > > the same time can also avoid
>> > > > conflicts
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > For lazy initialization:
>> > > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
>> > > itself
>> > > > > and
>> > > > > > > > > provide
>> > > > > > > > > > a
>> > > > > > > > > > > > > > > dedicated
>> > > > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > > implementation for lazy
>> loading,
>> > > such
>> > > > > as
>> > > > > > > > > > > > > > > ConfigurationCatalog
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > public class
>> ConfigurationCatalog
>> > > > > > implements
>> > > > > > > > > > > Catalog
>> > > > > > > > > > > > {
>> > > > > > > > > > > > > > > > > > > > > > >
>> ConfigurationCatalog(Map<String,
>> > > > > String>
>> > > > > > > > > > > properties)
>> > > > > > > > > > > > {
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > > }
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > I have added this design to
>> the
>> > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > Feng
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at
>> 10:03 AM
>> > > > Shammon
>> > > > > > FY <
>> > > > > > > > > > > > > > > > zjureel@gmail.com>
>> > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > I have no questions about
>> the
>> > > > > > functionality
>> > > > > > > > > of
>> > > > > > > > > > > > > > > > > `CatalogStore`,
>> > > > > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > different operations of
>> > multiple
>> > > > > > > > > > > `registerCatalog`
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > `storeCatalog`
>> > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
>> > > > implementation
>> > > > > in
>> > > > > > > > Trino
>> > > > > > > > > > is
>> > > > > > > > > > > > also
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > same,
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino
>> only
>> > > has
>> > > > > one
>> > > > > > > > > > > > > `createCatalog`,
>> > > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > save
>> > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > catalog to memory and
>> > > > `CatalogStore`.
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
>> > > > another
>> > > > > > > > > > > > > `registerCatalog`
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > `addCatalog`
>> > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > save a catalog in
>> `Map<String,
>> > > > > Catalog>
>> > > > > > > > > > catalogs`
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > `CatalogStore
>> > > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you
>> > mentioned
>> > > > > > before,
>> > > > > > > > the
>> > > > > > > > > > > > > > `Map<String,
>> > > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
>> > > > > > `CatalogManager`.
>> > > > > > > > How
>> > > > > > > > > > > about
>> > > > > > > > > > > > > > > saving
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog>
>> catalogs`
>> > > and
>> > > > > > > > > > `CatalogStore
>> > > > > > > > > > > > > > > > > catalogStore`
>> > > > > > > > > > > > > > > > > > > > > > together
>> > > > > > > > > > > > > > > > > > > > > > > > when it is registered or
>> added
>> > in
>> > > > > > > > > > > `CatalogManager`?
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
>> 5:22 PM
>> > > Feng
>> > > > > > Jin <
>> > > > > > > > > > > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply.
>> I
>> > am
>> > > > very
>> > > > > > sorry
>> > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > misunderstanding
>> > > > > > > > > > > > > > > > > > > > > > > caused
>> > > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
>> > > original
>> > > > > > > > > discussion.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
>> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
>> > > > pre-discussion
>> > > > > > [1]
>> > > > > > > > for
>> > > > > > > > > > > this
>> > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed
>> such a
>> > > > > > discussion
>> > > > > > > > > > before.
>> > > > > > > > > > > > > > > However,
>> > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > designing
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found
>> that
>> > > the
>> > > > > > logic of
>> > > > > > > > > > > > > > > CatalogManager
>> > > > > > > > > > > > > > > > > > itself
>> > > > > > > > > > > > > > > > > > > > > > doesn't
>> > > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We
>> > cannot
>> > > > > only
>> > > > > > > > > persist
>> > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > instances
>> > > > > > > > > > > > > > > > > > > > > > > > themselves*,
>> > > > > > > > > > > > > > > > > > > > > > > > > so exposing only
>> > > > > > registerCatalog(String
>> > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > > > might
>> > > > > > > > > > > > > > > > > > > > > > > > > not be enough to save
>> > Catalogs,
>> > > > > > because
>> > > > > > > > in
>> > > > > > > > > > the
>> > > > > > > > > > > > end
>> > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > still
>> > > > > > > > > > > > > > > > > > > need
>> > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > save
>> > > > > > > > > > > > > > > > > > > > > > > > > the configurations
>> > > corresponding
>> > > > to
>> > > > > > the
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > instances.
>> > > > > > > > > > > > > > > > > > > > > > Therefore,
>> > > > > > > > > > > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
>> > > > > CatalogStore
>> > > > > > > > > > interface
>> > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding
>> this
>> > > > part, I
>> > > > > > also
>> > > > > > > > > took
>> > > > > > > > > > > > > > > inspiration
>> > > > > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > > > > Trino's
>> > > > > > > > > > > > > > > > > > > > > > > > > implementation[1].
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > @yuxia
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
>> > handling
>> > > > > > catalog
>> > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > name
>> > > > > > > > > > > > > > > > > > > looks a
>> > > > > > > > > > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > complex to me.
>> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the
>> > suggestion. I
>> > > > > will
>> > > > > > > > > provide
>> > > > > > > > > > a
>> > > > > > > > > > > > > > detailed
>> > > > > > > > > > > > > > > > > > > > description
>> > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > code examples for this
>> part,
>> > > and
>> > > > > add
>> > > > > > it
>> > > > > > > > to
>> > > > > > > > > > the
>> > > > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
>> > > > > > `addCatalog`
>> > > > > > > > > still
>> > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > confused
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > me.
>> > > > > > > > > > > > > > > > > > > > > > > IIUC,
>> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
>> > > > catalog
>> > > > > to
>> > > > > > > > > > > > CatalogStore,
>> > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > > > > > > renaming
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
>> very
>> > > > > personal
>> > > > > > > > > > opinion,
>> > > > > > > > > > > > you
>> > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > decide
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > take
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
>> > > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
>> > > intuitive
>> > > > > to
>> > > > > > me.
>> > > > > > > > I
>> > > > > > > > > > > don't
>> > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > any
>> > > > > > > > > > > > > > > > > > > problem
>> > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > it.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
>> > > statement,
>> > > > > > which
>> > > > > > > > > > method
>> > > > > > > > > > > > will
>> > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > called?
>> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
>> > > > `addCatalog`?
>> > > > > > I'm
>> > > > > > > > > > > wondering
>> > > > > > > > > > > > > > > whether
>> > > > > > > > > > > > > > > > > user
>> > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > add a
>> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
>> > > > stement.
>> > > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my
>> > original
>> > > > > > design
>> > > > > > > > was
>> > > > > > > > > to
>> > > > > > > > > > > add
>> > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > directly
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this
>> would
>> > > > > disrupt
>> > > > > > the
>> > > > > > > > > > > existing
>> > > > > > > > > > > > > > > logic.
>> > > > > > > > > > > > > > > > > > > > > Therefore, I
>> > > > > > > > > > > > > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
>> > > > > > configuration to
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > initialize
>> > > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at
>> the
>> > > same
>> > > > > time
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really
>> neccessary
>> > to
>> > > > > > provide a
>> > > > > > > > > > > default
>> > > > > > > > > > > > > > > > > > implmentation
>> > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > interface
>> > > `CatalogStoreFactory`?
>> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
>> > > > otherwise
>> > > > > we
>> > > > > > > > would
>> > > > > > > > > > > need
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > introduce
>> > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > > > additional
>> > > > > > > > > > > > > > > > > > > > > > > > > Map to store the
>> > configuration
>> > > > for
>> > > > > > lazy
>> > > > > > > > > > > loading.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
>> > > > > registration
>> > > > > > for
>> > > > > > > > > > > catalog.
>> > > > > > > > > > > > > > > > > > > > > > > > > I don't think
>> > > > > registerCatalog(String
>> > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
>> > > > interface
>> > > > > > > > because
>> > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > already
>> > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > > > instance.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
>> > > > > > initialization
>> > > > > > > > > for
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > than
>> > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
>> > > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
>> > > > > > inaccurate. It
>> > > > > > > > > > > should
>> > > > > > > > > > > > be
>> > > > > > > > > > > > > > > lazy
>> > > > > > > > > > > > > > > > > > > > > registration
>> > > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
>> > > > > > registration. I
>> > > > > > > > > have
>> > > > > > > > > > > > > already
>> > > > > > > > > > > > > > > > > updated
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > title
>> > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > the FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > [1].
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
>> > 4:27 PM
>> > > > > > Shammon
>> > > > > > > > FY <
>> > > > > > > > > > > > > > > > > zjureel@gmail.com
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
>> > > > > > `registerCatalog`
>> > > > > > > > > > > method
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > `CatalogManager`,
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
>> > > method
>> > > > > > named
>> > > > > > > > > > > > `addCatalog`
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > `storeCatalog`
>> > > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > added.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned,
>> the
>> > > > memory
>> > > > > > > > catalog
>> > > > > > > > > > is a
>> > > > > > > > > > > > > > > `cache`,
>> > > > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > concept
>> > > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
>> > > > exposed
>> > > > > to
>> > > > > > > > > users.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
>> > > > pre-discussion
>> > > > > > [1]
>> > > > > > > > for
>> > > > > > > > > > > this
>> > > > > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > > > > > Please
>> > > > > > > > > > > > > > > > > > > > > correct
>> > > > > > > > > > > > > > > > > > > > > > > me
>> > > > > > > > > > > > > > > > > > > > > > > > if
>> > > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
>> > > conclusion
>> > > > > of
>> > > > > > that
>> > > > > > > > > > > > > discussion
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
>> > > > interface
>> > > > > > and
>> > > > > > > > > > > implement
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > different
>> > > > > > > > > > > > > > > > > > > > > > > stores
>> > > > > > > > > > > > > > > > > > > > > > > > > such
>> > > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and
>> > external
>> > > > > > system.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
>> > > between
>> > > > > the
>> > > > > > > > > current
>> > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > design
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > > > > > > > > conclusion.
>> > > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal
>> of
>> > > the
>> > > > > > > > discussion
>> > > > > > > > > > in
>> > > > > > > > > > > > > thread
>> > > > > > > > > > > > > > > > [1] ?
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > [1]
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
>> > > 3:41 PM
>> > > > > > yuxia <
>> > > > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
>> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for
>> driving
>> > > this
>> > > > > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
>> > > handling
>> > > > > > catalog
>> > > > > > > > > with
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > same
>> > > > > > > > > > > > > > > > name
>> > > > > > > > > > > > > > > > > > > > looks a
>> > > > > > > > > > > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
>> > > it'll
>> > > > be
>> > > > > > > > better
>> > > > > > > > > to
>> > > > > > > > > > > > > explain
>> > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > java
>> > > > > > > > > > > > > > > > > > > > > > > doc
>> > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > these methods and
>> give a
>> > > > brief
>> > > > > > > > example
>> > > > > > > > > in
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > > > FLIP.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method
>> name
>> > > > > > `addCatalog`
>> > > > > > > > > > still
>> > > > > > > > > > > > > looks
>> > > > > > > > > > > > > > > > > confused
>> > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > me.
>> > > > > > > > > > > > > > > > > > > > > > > > IIUC,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > this method is for
>> > storing
>> > > > > > catalog to
>> > > > > > > > > > > > > > CatalogStore,
>> > > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > > > > > > > > renaming
>> > > > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`?
>> It's
>> > > very
>> > > > > > personal
>> > > > > > > > > > > > opinion,
>> > > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > decide
>> > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > take
>> > > > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
>> > > > > statement,
>> > > > > > > > which
>> > > > > > > > > > > method
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > called?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
>> > > > > > `addCatalog`?
>> > > > > > > > I'm
>> > > > > > > > > > > > > wondering
>> > > > > > > > > > > > > > > > > whether
>> > > > > > > > > > > > > > > > > > > user
>> > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > > add a
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with
>> SQL
>> > > > > > stement.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really
>> > neccessary
>> > > to
>> > > > > > > > provide a
>> > > > > > > > > > > > default
>> > > > > > > > > > > > > > > > > > > implmentation
>> > > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > > > interface
>> > > > > `CatalogStoreFactory`?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
>> > > > > > registration
>> > > > > > > > for
>> > > > > > > > > > > > catalog.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a
>> catalog
>> > > > with
>> > > > > > CREATE
>> > > > > > > > > > > > CATALOG,
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
>> > used
>> > > > by
>> > > > > > > > default.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
>> > > registration
>> > > > is
>> > > > > > the
>> > > > > > > > > > default
>> > > > > > > > > > > > > > > behavior,
>> > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > > > > > > > > any
>> > > > > > > > > > > > > > > > > > > > > > > > way
>> > > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch
>> to
>> > > > > > synchronous
>> > > > > > > > > > > > > registration
>> > > > > > > > > > > > > > > just
>> > > > > > > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > > > > > > > > before?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Will both method
>> > > `addCatalog`
>> > > > > and
>> > > > > > > > > > > > > > `registerCatalog`
>> > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
>> > > > > > registration,
>> > > > > > > > it
>> > > > > > > > > > may
>> > > > > > > > > > > > well
>> > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > CREATE
>> > > > > > > > > > > > > > > > > > > > > > CATALOG
>> > > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully,
>> > but
>> > > > then
>> > > > > > the
>> > > > > > > > > > > following
>> > > > > > > > > > > > > > > CREATE
>> > > > > > > > > > > > > > > > > > TABLE
>> > > > > > > > > > > > > > > > > > > > > > > statement
>> > > > > > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog
>> fail
>> > > to
>> > > > > > open.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break
>> > change
>> > > > > which
>> > > > > > > > > should
>> > > > > > > > > > be
>> > > > > > > > > > > > > > > > highlighted
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > > > > > FLIP,
>> > > > > > > > > > > > > > > > > > > > > > > > > may
>> > > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility
>> part.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying
>> > asynchronous
>> > > > > > > > > > registration, I
>> > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > expect
>> > > > > > > > > > > > > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to
>> > open
>> > > > or
>> > > > > > > > register
>> > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > background,
>> > > > > > > > > > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > > > > > > > > > your previous
>> comments,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance
>> > will
>> > > be
>> > > > > > > > > initialized
>> > > > > > > > > > > if
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > been
>> > > > > > > > > > > > > > > > > > > > > > > > > initialized
>> > > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the
>> > initialization
>> > > > > > process
>> > > > > > > > > fails,
>> > > > > > > > > > > > these
>> > > > > > > > > > > > > > > > > > statements
>> > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > > > > executed
>> successfully."
>> > > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like
>> lazy
>> > > > > > > > initialization
>> > > > > > > > > > for
>> > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > than
>> > > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
>> > > > > > > > jinfeng1995@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
>> > > > > dev@flink.apache.org
>> > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月
>> 03日
>> > 下午
>> > > > > > 3:27:45
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP
>> > 295:
>> > > > > > Support
>> > > > > > > > > > > > persistence
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
>> > > registration
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you
>> > all
>> > > > for
>> > > > > > your
>> > > > > > > > > > > interest
>> > > > > > > > > > > > in
>> > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > DISCUSS.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a
>> catalog
>> > > > with
>> > > > > > the
>> > > > > > > > same
>> > > > > > > > > > > name
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > them?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a
>> > crucial
>> > > > > > point.
>> > > > > > > > > Based
>> > > > > > > > > > on
>> > > > > > > > > > > > my
>> > > > > > > > > > > > > > > > current
>> > > > > > > > > > > > > > > > > > > > personal
>> > > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
>> > > > Map<String,
>> > > > > > > > Catalog>
>> > > > > > > > > > > > > catalogs
>> > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > serve
>> > > > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > > > > cache
>> > > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated
>> catalogs
>> > > and
>> > > > > > have
>> > > > > > > > the
>> > > > > > > > > > > > highest
>> > > > > > > > > > > > > > > > > priority.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > There are three
>> methods
>> > > that
>> > > > > can
>> > > > > > > > affect
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1.
>> registerCatalog(String
>> > > > > > > > catalogName,
>> > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
>> > > catalog
>> > > > > > instance
>> > > > > > > > > > into
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2.
>> > unregisterCatalog(String
>> > > > > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes
>> the
>> > > > catalog
>> > > > > > > > > instance
>> > > > > > > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > catalogName
>> > > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
>> > > Catalog>
>> > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
>> > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method first
>> > retrieves
>> > > > the
>> > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String,
>> Catalog>
>> > > > > > catalogs. If
>> > > > > > > > > the
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > does
>> > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > exist,
>> > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the
>> > corresponding
>> > > > > > > > > configuration
>> > > > > > > > > > > > from
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > CatalogStore,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and
>> puts
>> > > the
>> > > > > > > > > initialized
>> > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > into
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
>> > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > The following two
>> methods
>> > > > only
>> > > > > > modify
>> > > > > > > > > the
>> > > > > > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
>> > > > > catalogName,
>> > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > String>
>> > > > > > > > > > > > > > > > > > > > > properties)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
>> > > > > properties
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > > > > > > > catalogStore
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > checks
>> > > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
>> > > > catalogName
>> > > > > > with
>> > > > > > > > the
>> > > > > > > > > > > same
>> > > > > > > > > > > > > > name.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2.
>> removeCatalog(String
>> > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes
>> the
>> > > > > specified
>> > > > > > > > > > > > configuration
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > specified
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
>> > > > > catalogStore.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > The following are
>> > possible
>> > > > > > conflict
>> > > > > > > > > > > > scenarios:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the
>> corresponding
>> > > > > > catalogName
>> > > > > > > > > > > already
>> > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not
>> in
>> > the
>> > > > > > > > Map<String,
>> > > > > > > > > > > > > Catalog>,
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
>> > > > > > catalogName,
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be
>> directly
>> > > saved
>> > > > > to
>> > > > > > the
>> > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the
>> corresponding
>> > > > > > catalogName
>> > > > > > > > > > > already
>> > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
>> > > > > Map<String,
>> > > > > > > > > > Catalog>,
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
>> > > catalog)
>> > > > > > method
>> > > > > > > > > will
>> > > > > > > > > > > > fail.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the
>> corresponding
>> > > > > > catalogName
>> > > > > > > > > > > already
>> > > > > > > > > > > > > > > exists
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>,
>> the
>> > > > > > > > > > addCatalog(String
>> > > > > > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties)
>> > method
>> > > > can
>> > > > > > > > directly
>> > > > > > > > > > > save
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > properties
>> > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
>> > > > > > > > getCatalog(String
>> > > > > > > > > > > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties
>> > for
>> > > > > > > > > initialization
>> > > > > > > > > > > > > because
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance
>> already
>> > > > exists
>> > > > > > in
>> > > > > > > > > > catalogs
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > prioritized.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
>> > > > > > > > > > > unregisterCatalog(String
>> > > > > > > > > > > > > > > > > > catalogName)
>> > > > > > > > > > > > > > > > > > > > > method
>> > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
>> > > > > > corresponding to
>> > > > > > > > > the
>> > > > > > > > > > > > > original
>> > > > > > > > > > > > > > > > > > > catalogName
>> > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > > necessary.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will
>> confuse
>> > > > users
>> > > > > > that
>> > > > > > > > > > > > > > > > > > > `registerCatalog(String
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
>> > > catalog)`
>> > > > > in
>> > > > > > the
>> > > > > > > > > > > > > `Map<String,
>> > > > > > > > > > > > > > > > > > Catalog>
>> > > > > > > > > > > > > > > > > > > > > > > catalogs`
>> > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> `registerCatalog(String
>> > > > > > catalogName,
>> > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > String>
>> > > > > > > > > > > > > > > > > > > > > > > properties)
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially
>> > lead
>> > > > to
>> > > > > > > > > > confusion. I
>> > > > > > > > > > > > > > suggest
>> > > > > > > > > > > > > > > > > > > changing
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps
>> to
>> > > > > > > > > addCatalog(String
>> > > > > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > > > > > > Map<String,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties),
>> as
>> > > > > > previously
>> > > > > > > > > > > mentioned
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > @Hang
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > add
>> > > `registerCatalog(String
>> > > > > > > > > > > > > catalogName,Catalog
>> > > > > > > > > > > > > > > > > > catalog,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)`
>> method
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is
>> > already
>> > > an
>> > > > > > > > instance,
>> > > > > > > > > > > > adding
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > "lazyInit"
>> > > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
>> > > > > > > > registerCatalog(String
>> > > > > > > > > > > > > > > catalogName,
>> > > > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > > catalog)
>> > > > > > > > > > > > > > > > > > > > > > > > > > > method may not
>> > necessarily
>> > > > > > result in
>> > > > > > > > > lazy
>> > > > > > > > > > > > > > > > > initialization.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think
>> > about
>> > > > > > > > encryption
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
>> > > > > necessary,
>> > > > > > but
>> > > > > > > > > > > perhaps
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > encryption
>> > > > > > > > > > > > > > > > > > > > > logic
>> > > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented
>> in
>> > > the
>> > > > > > > > > CatalogStore
>> > > > > > > > > > > > when
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > actually
>> > > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance,
>> the
>> > > > > > > > > FileCatalogStore
>> > > > > > > > > > > > could
>> > > > > > > > > > > > > > > > encrypt
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > > > > > > when
>> > > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
>> > > while
>> > > > > the
>> > > > > > > > > > > > > > MemoryCatalogStore
>> > > > > > > > > > > > > > > > > does
>> > > > > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > > > > > > require
>> > > > > > > > > > > > > > > > > > > > > > > > > > > encryption.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need
>> the
>> > > > > > > > > > > > `MemoryCatalogStore`?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is
>> necessary
>> > to
>> > > > > have a
>> > > > > > > > > > > > > > > MemoryCatalogStore
>> > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > default
>> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which
>> > saves
>> > > > > > Catalog
>> > > > > > > > > > > > > > configurations
>> > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > memory.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want
>> to
>> > > > > > implement
>> > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > loading
>> > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > Catalog,
>> > > > > > > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > > > > > > > would need to
>> introduce
>> > > > > > additional
>> > > > > > > > > cache.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
>> > > > > > registration
>> > > > > > > > > > failed?
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This is also a
>> critical
>> > > > > concern.
>> > > > > > > > When
>> > > > > > > > > > > > > executing
>> > > > > > > > > > > > > > > DDL,
>> > > > > > > > > > > > > > > > > > DQL,
>> > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > DML
>> > > > > > > > > > > > > > > > > > > > > > > > > > > statements that
>> > reference a
>> > > > > > specific
>> > > > > > > > > > > Catalog,
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > > > > instance
>> > > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized
>> if it
>> > > has
>> > > > > not
>> > > > > > > > been
>> > > > > > > > > > > > > > initialized
>> > > > > > > > > > > > > > > > yet.
>> > > > > > > > > > > > > > > > > > If
>> > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > initialization process
>> > > fails,
>> > > > > > these
>> > > > > > > > > > > > statements
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > executed
>> > > > > > > > > > > > > > > > > > > > > > > > > > > successfully.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
>> > > > Catalog>
>> > > > > > > > catalogs
>> > > > > > > > > > and
>> > > > > > > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > > > > > > catalogstore
>> > > > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all
>> > have
>> > > > the
>> > > > > > same
>> > > > > > > > > > > catalog
>> > > > > > > > > > > > > > name,
>> > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > > correponding
>> > > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
>> > > instances.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be
>> > resolved
>> > > by
>> > > > > > > > referring
>> > > > > > > > > > to
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > previous
>> > > > > > > > > > > > > > > > > > > > > > responses.
>> > > > > > > > > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that
>> the
>> > > > > Catalog
>> > > > > > > > that
>> > > > > > > > > > has
>> > > > > > > > > > > > been
>> > > > > > > > > > > > > > > most
>> > > > > > > > > > > > > > > > > > > recently
>> > > > > > > > > > > > > > > > > > > > > > used
>> > > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority
>> > for
>> > > > > > subsequent
>> > > > > > > > > > use.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
>> > > > 10:58 PM
>> > > > > > > > Xianxun
>> > > > > > > > > > Ye
>> > > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving
>> this
>> > > > Flip,
>> > > > > > I do
>> > > > > > > > > > > believe
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > Flip
>> > > > > > > > > > > > > > > > > > > could
>> > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > helpful
>> > > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I
>> > work
>> > > > for
>> > > > > > also
>> > > > > > > > > > > manages a
>> > > > > > > > > > > > > lot
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > catalogs,
>> > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks
>> > becomes
>> > > > > slow
>> > > > > > > > > because
>> > > > > > > > > > of
>> > > > > > > > > > > > > > > loading a
>> > > > > > > > > > > > > > > > > > > number
>> > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the
>> catalogs in
>> > > the
>> > > > > > user's
>> > > > > > > > > > Flink
>> > > > > > > > > > > > SQL
>> > > > > > > > > > > > > > > > through
>> > > > > > > > > > > > > > > > > > > > regular
>> > > > > > > > > > > > > > > > > > > > > > > > > expression
>> > > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
>> > > catalogs
>> > > > > to
>> > > > > > > > > improve
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > speed
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > task
>> > > > > > > > > > > > > > > > > > > > > > > > submission.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > After reading
>> flip-295, I
>> > > > have
>> > > > > > some
>> > > > > > > > > > > > questions:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
>> > > catalog
>> > > > > with
>> > > > > > > > > CREATE
>> > > > > > > > > > > > > CATALOG,
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
>> > used
>> > > > by
>> > > > > > > > default.
>> > > > > > > > > > > What
>> > > > > > > > > > > > if
>> > > > > > > > > > > > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > > > > > > > > > > registration
>> > > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
>> > > implement
>> > > > > > > > > asynchronous
>> > > > > > > > > > > > > > > > registration?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the
>> same
>> > > > > > question
>> > > > > > > > with
>> > > > > > > > > > > > > Shammon’s
>> > > > > > > > > > > > > > > > first
>> > > > > > > > > > > > > > > > > > > > > comment.
>> > > > > > > > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
>> > > catalogs
>> > > > > and
>> > > > > > > > > > > > CatalogStore
>> > > > > > > > > > > > > > > > > > catalogstore
>> > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all
>> have
>> > the
>> > > > > same
>> > > > > > > > > catalog
>> > > > > > > > > > > > name,
>> > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > correponding
>> > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
>> > > instances.
>> > > > > For
>> > > > > > > > > example:
>> > > > > > > > > > > > > > > catalog1->
>> > > > > > > > > > > > > > > > > > > > > jdbcCatalog
>> > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but
>> catalog1 ->
>> > > > > > hiveCatalog
>> > > > > > > > > in
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > catalogstore.
>> > > > > > > > > > > > > > > > > > > > > Would
>> > > > > > > > > > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > > > > > > > > case
>> > > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we
>> should
>> > > > define
>> > > > > > > > > `catalogs`
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > `catalogstore`
>> > > > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > > > > > clearly.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日
>> 11:13，Hang
>> > > > Ruan
>> > > > > <
>> > > > > > > > > > > > > > > > ruanhang1993@gmail.com
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > 写道：
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Samrat,
Thanks for your advice.
> 1. The createCatalog method does not mention any exceptions being thrown.
CreateCatalog will throw CatalogException like registerCatalog. As
CatalogException is a RuntimeException,
there is no explicit declaration of throwing Exceptions in CatalogManager
and TableEnvironment.
To avoid misunderstandings, I have added the "throw CatalogException" flag
to the createCatalog method definition of CatalogStore.
> 2. Could you please provide an exhaustive list of the supported kinds?
Sure, the documentation now includes the configuration of the built-in
CatalogStore as well as how to configure a custom CatalogStore.
Best,
Feng
On Sun, Jun 4, 2023 at 4:23 AM Samrat Deb <de...@gmail.com> wrote:
> Hi Feng,
>
> Thank you for providing the proposal. I believe this feature will be highly
> valuable.
>
> I have a couple of inquiries:
>
> 1. According to the documentation [1], the createCatalog method does not
> mention any exceptions being thrown. However, I would like to confirm if it
> is always true that there will be no failures in all scenarios. Please let
> me know if there is any additional information I may have missed.
>
> 2. Regarding the registration process using the `table.catalog-store.kind`
> configuration, could you please provide an exhaustive list of the supported
> kinds?
> It would be great to have a comprehensive understanding of the options
> available.
>
> Bests,
> Samrat
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
> On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
>
> > Hi Hang and Jark
> >
> > Thank you very much for your reply.
> >
> > @Jark
> > > 1. Could you move the CatalogStore registration API to the "Public
> > Interface" section?
> >
> > Indeed, it is more reasonable.
> >
> > > 2. We should prefix "table." for the CatalogStore configuration.
> >
> > Sure, the names that you provided are indeed more logical and easier to
> > understand.
> >
> > > 3. About the open/close method in CatalogStoreFactory.
> >
> > I also agree with the proposed requirement scenario and design. I have
> > already made modifications to the interface.
> >
> > @Hang
> >
> > > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> > the resource.
> >
> > I have already added the missing method.
> >
> > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >
> > Sorry for the misspelling, I updated this part.
> >
> > > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
> > to init or close its resource
> >
> > This is indeed a scenario that needs to be considered, and I have added
> > relevant methods accordingly.
> > Additionally, I have included some relevant description information in
> the
> > documentation to help others understand
> > why the open/close methods need to be added.
> >
> >
> > Best,
> > Feng
> >
> > On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >
> > > Thank you for the update.
> > >
> > > 1. Could you move the CatalogStore registration API to the "Public
> > > Interface" section?
> > > "Proposed Changes" is more like a place to describe the implementation
> > > details.
> > >
> > > 2. We should prefix "table." for the CatalogStore configuration.
> Besides,
> > > the config key
> > > name should be hierarchical[1]. Therefore, it may be better to use:
> > > "table.catalog-store.kind"
> > > "table.catalog-store.file.path"
> > >
> > > 3. I think Hang's suggestions make sense.
> > >
> > > Others look good to me.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
> > >
> > > > Hi, Feng.
> > > >
> > > > Thanks for the update.
> > > > The current design looks good to me. I have some minor comments.
> > > >
> > > > 1. The `CatalogStore` need the `open`/`close` methods to init or
> close
> > > the
> > > > resource. For example, when storing the information in MySQL, the
> store
> > > > needs to open and close the connections.
> > > >
> > > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > > >
> > > > 3. About the usage in the sql gateway.
> > > > Considering the usage in sql gateway, the sql gateway may create a
> > > > CatalogStore for each session.
> > > > If we are using the MySqlCatalogStore, there would be so many
> > > connections.
> > > > How can we reuse the connection among these sessions?
> > > > I think sql gateway need to maintain a connection pool in
> > > > the CatalogStoreFactory and each session get its own connection from
> > the
> > > > pool when it is created.
> > > > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> > > init
> > > > or close its resource.
> > > > Or is there a better way?
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > > >
> > > > > Thanks Jingsong.
> > > > >
> > > > > > Just naming, maybe `createCatalog` in TableEnv
> > > > >
> > > > > +1 For this, I have already updated FLIP.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > >
> > > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <
> jingsonglee0@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Feng,
> > > > > >
> > > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > > > functions are converted to createxxx from registerxxx.
> > > > > >
> > > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi jark, thanks for your suggestion.
> > > > > > >
> > > > > > > > 1. How to register the CatalogStore for Table API? I think
> the
> > > > > > > CatalogStore should be immutable once TableEnv is created.
> > > Otherwise,
> > > > > > there
> > > > > > > might be some data inconsistencies when CatalogStore is
> changed.
> > > > > > >
> > > > > > > Yes, We should initialize the CatalogStore when creating the
> > > > TableEnv.
> > > > > > > Therefore, my current proposal is to add a method to configure
> > the
> > > > > > > CatalogStore in EnvironmentSettings.
> > > > > > >
> > > > > > > // Initialize a catalog Store
> > > > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > > > >
> > > > > > > // set up the Table API
> > > > > > > final EnvironmentSettings settings =
> > > > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > > > .withCatalogStore(catalogStore)
> > > > > > > .build();
> > > > > > >
> > > > > > > final TableEnvironment tableEnv =
> > > > > > TableEnvironment.create(settings);
> > > > > > >
> > > > > > >
> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> > default
> > > > > > method,
> > > > > > > not an interface method?
> > > > > > >
> > > > > > > Sorry, While I did refer to the Catalog interface, I agree that
> > as
> > > a
> > > > > new
> > > > > > > interface, the CatalogStoreFactory should not have a default
> > method
> > > > but
> > > > > > an
> > > > > > > interface method. I have already modified the interface.
> > > > > > >
> > > > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > > > deprecated`registerCatalog`.
> > > > > > > > 4. In the "Compatibility" section, would be better to mention
> > the
> > > > > > changed
> > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> > other
> > > > > > > persisted catalog store) is used.
> > > > > > >
> > > > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > > > >
> > > > > > >
> > > > > > > [1].
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > > > The current design looks pretty good to me. I just have some
> > > minor
> > > > > > > > comments.
> > > > > > > >
> > > > > > > > 1. How to register the CatalogStore for Table API? I think
> the
> > > > > > CatalogStore
> > > > > > > > should be immutable once TableEnv is created. Otherwise,
> there
> > > > might
> > > > > be
> > > > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > > > >
> > > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> > default
> > > > > > method,
> > > > > > > > not an interface method?
> > > > > > > >
> > > > > > > > 3. Please mention the alternative API in Javadoc for the
> > > deprecated
> > > > > > > > `registerCatalog`.
> > > > > > > >
> > > > > > > > 4. In the "Compatibility" section, would be better to mention
> > the
> > > > > > changed
> > > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> > > other
> > > > > > > > persisted catalog store) is used.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jark
> > > > > > > >
> > > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <jinfeng1995@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > > > >
> > > > > > > > > @Ron
> > > > > > > > >
> > > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore
> > method,
> > > > do
> > > > > we
> > > > > > > > need
> > > > > > > > > to provide a default implementation?
> > > > > > > > >
> > > > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory
> to
> > > > > create
> > > > > > an
> > > > > > > > > InMemoryCatalogStore.
> > > > > > > > >
> > > > > > > > > > If we get a Catalog from CatalogStore, after
> initializing
> > > it,
> > > > > > whether
> > > > > > > > we
> > > > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > > > >
> > > > > > > > > Yes, in the current design, catalogs are stored as
> snapshots,
> > > and
> > > > > > once
> > > > > > > > > initialized, the Catalog will be placed in the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs.
> > > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > > > primary
> > > > > > > > source
> > > > > > > > > for obtaining the corresponding Catalog.
> > > > > > > > >
> > > > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > > > `catalog.store.path`?
> > > > > > > > >
> > > > > > > > > I think it is okay. Adding "sql" at the beginning may seem
> a
> > > bit
> > > > > > > > strange. I
> > > > > > > > > will update the FLIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > @Shammon
> > > > > > > > >
> > > > > > > > > Thank you for the review. I have made the necessary
> > > corrections.
> > > > > > > > > Regarding the modifications made to the Public Interface, I
> > > have
> > > > > also
> > > > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> > zjureel@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > > > >
> > > > > > > > > > 1. The modification of `CatalogManager` should not be in
> > > > `Public
> > > > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > > > >
> > > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore`
> and
> > > > > > > > > > `CatalogStoreFactory`
> > > > > > > > > >
> > > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> > > should
> > > > be
> > > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > > > catalogStore.get(catalogName);`
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shammon FY
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <
> > ron9.liu@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Feng
> > > > > > > > > > >
> > > > > > > > > > > Thanks for driving this FLIP, this proposal is very
> > useful
> > > > for
> > > > > > > > catalog
> > > > > > > > > > > management.
> > > > > > > > > > > I have some small questions:
> > > > > > > > > > >
> > > > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > > > method,
> > > > > > do we
> > > > > > > > > > need
> > > > > > > > > > > to provide a default implementation?
> > > > > > > > > > > 2. If we get Catalog from CatalogStore, after
> > initializing
> > > > it,
> > > > > > > > whether
> > > > > > > > > we
> > > > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > > > `sql.catalog.store.file.path`, how about renaming them
> to
> > > > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一
> 21:19写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > > But from the code in Proposed Changes, once we
> > > register
> > > > > the
> > > > > > > > > Catalog,
> > > > > > > > > > > we
> > > > > > > > > > > > initialize it and open it. right?
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > > > original
> > > > > > > > CREATE
> > > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > > > registerCatalog
> > > > > > > > > so
> > > > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > > > >
> > > > > > > > > > > > In the current design, lazy initialization is mainly
> > > > > reflected
> > > > > > in
> > > > > > > > > > > > getCatalog. If CatalogStore has already saved some
> > > catalog
> > > > > > > > > > > configurations,
> > > > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > > > luoyuxia@alumni.sjtu.edu.cn
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > > > initialization*.
> > > > > > > > > If
> > > > > > > > > > > i'm
> > > > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > > > >
> > > > > > > > > > > > > IIUC, lazy initialization means only you need to
> > access
> > > > the
> > > > > > > > > catalog,
> > > > > > > > > > > then
> > > > > > > > > > > > > you initialize it. But from the code in Proposed
> > > Changes,
> > > > > > once we
> > > > > > > > > > > > register
> > > > > > > > > > > > > the Catalog,
> > > > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > >
> > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > > Catalog
> > > > > > > > > > > configuration
> > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One of the major changes is that current design
> will
> > > > > provide
> > > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead
> > of a
> > > > > > cache,
> > > > > > > > > which
> > > > > > > > > > > > means
> > > > > > > > > > > > > once it has been initialized, any changes done by
> > other
> > > > > > sessions
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > affect it. Point 6 described follow-up options for
> > > > further
> > > > > > > > > > improvement.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all, I would like to update you on the latest
> > > > progress
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon
> > FY,
> > > > > > ShengKai
> > > > > > > > > Fang
> > > > > > > > > > > > and I
> > > > > > > > > > > > > > had an offline discussion regarding the overall
> > > > solution
> > > > > > for
> > > > > > > > > Flink
> > > > > > > > > > > > > > CatalogStore. We have reached a consensus and I
> > have
> > > > > > updated
> > > > > > > > the
> > > > > > > > > > > final
> > > > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > > > configuration
> > > > > > > > > > similar
> > > > > > > > > > > > to
> > > > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > > > 2.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > > > storeCatalog(String
> > > > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > > > CatalogDescriptor
> > > > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > > > will both use CatalogDescriptor instead of
> > Catalog
> > > > > > instance.
> > > > > > > > > > This
> > > > > > > > > > > > way,
> > > > > > > > > > > > > > CatalogStore will only be responsible for
> saving
> > > and
> > > > > > > > > retrieving
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > configurations without having to initialize
> > > > catalogs.
> > > > > > > > > > > > > > 3.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The default registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > > > function in CatalogManager will be marked as
> > > > > deprecated.
> > > > > > > > > > > > > > 4.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > A new function registerCatalog(String
> > catalogName,
> > > > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > > > catalog) will be added to serve as the default
> > > > > > registration
> > > > > > > > > > > function
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > > > 5.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager
> > will
> > > > > > remain
> > > > > > > > > > > unchanged
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > save initialized catalogs.This means that
> > deletion
> > > > > > > > operations
> > > > > > > > > > from
> > > > > > > > > > > > one
> > > > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > > > 6.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > To support multi-session synchronization
> > scenarios
> > > > for
> > > > > > > > > deletions
> > > > > > > > > > > > later
> > > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > > > configurable.There
> > > > > > > > > > > may
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > three possible situations:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > > > automatically
> > > > > > > > > clear
> > > > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > c.No caching of any instances; each call to
> > > > getCatalog
> > > > > > > > > creates a
> > > > > > > > > > > new
> > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you very much for your attention and
> > > suggestions
> > > > on
> > > > > > this
> > > > > > > > > > > FLIP. A
> > > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions
> on
> > > the
> > > > > > entire
> > > > > > > > > > design
> > > > > > > > > > > > and
> > > > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If you have any further suggestions or feedback
> > about
> > > > > this
> > > > > > FLIP
> > > > > > > > > > > please
> > > > > > > > > > > > > feel
> > > > > > > > > > > > > > free to share.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to
> > me.
> > > > We
> > > > > > could
> > > > > > > > > > still
> > > > > > > > > > > > > > > reconsider in the future how to provide more
> > common
> > > > > > built-in
> > > > > > > > > > cache
> > > > > > > > > > > > > > > functionality in CatalogManager so that not
> every
> > > > > > > > CatalogSotre
> > > > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There might be more such issues. I would
> > > suggest
> > > > > you
> > > > > > > > > > > completely
> > > > > > > > > > > > > walk
> > > > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am very sorry for my carelessness and at
> the
> > > same
> > > > > > time, I
> > > > > > > > > > > greatly
> > > > > > > > > > > > > > > > appreciate your careful review. I have
> > thoroughly
> > > > > > checked
> > > > > > > > the
> > > > > > > > > > > > entire
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If I am not mistaken, with the current
> FLIP
> > > > > design,
> > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > could work without Optional CatalogStore
> being
> > > > > > configured.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yes, in the original design, CatalogStore was
> > not
> > > > > > necessary
> > > > > > > > > > > because
> > > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog>
> > catalogs
> > > > to
> > > > > > store
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > > > Therefore, I
> > > > > > > > > > modified
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > of the design and removed Map<String,
> Catalog>
> > > > > catalogs
> > > > > > > > from
> > > > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve
> > as
> > > > the
> > > > > > > > default
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > > > functionality of
> > > > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > > > The previous plan that kept
> > > > > > Map<String,Catalog>catalogs has
> > > > > > > > > > been
> > > > > > > > > > > > > moved
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are still many places contain
> > > inconsistent
> > > > > > content,
> > > > > > > > > > e.g.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still
> used.
> > > > > > > > > > > > > > > > > 2. The java comment of the method
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> > > tell
> > > > > > what the
> > > > > > > > > > > method
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There might be more such issues. I would
> > > suggest
> > > > > you
> > > > > > > > > > completely
> > > > > > > > > > > > > walk
> > > > > > > > > > > > > > > > > through the FLIP again and fix those
> issues.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> > > that
> > > > > you
> > > > > > will
> > > > > > > > > > build
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > functionality in the CatalogStore? This is
> a
> > > very
> > > > > > > > different
> > > > > > > > > > > > design
> > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > from what the current FLIP described. If I
> am
> > > not
> > > > > > > > mistaken,
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > current FLIP design, CatalogManager could
> > work
> > > > > > without
> > > > > > > > > > Optional
> > > > > > > > > > > > > > > > > CatalogStore being configured. That is the
> > > reason
> > > > > > why I
> > > > > > > > > > > mentioned
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > last email that the example code wrt the
> > > Optional
> > > > > > > > > > CatalogStore
> > > > > > > > > > > is
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are still some NIT issues in the
> > FLIP
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you very much for the careful
> > review. I
> > > > > have
> > > > > > > > > already
> > > > > > > > > > > made
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > > > multi-instance
> > > > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Currently, there are conflicts in
> multiple
> > > > > > scenarios
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > design. I am thinking whether we should
> > > remove
> > > > > > > > > 'Map<String,
> > > > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > > > InMemoryCatalogStore.
> > > > > > > > > > This
> > > > > > > > > > > > > way,
> > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > can implement their own CatalogStore to
> > > achieve
> > > > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you
> > think?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There
> are
> > > > still
> > > > > > some
> > > > > > > > > NIT
> > > > > > > > > > > > issues
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore
> > has
> > > > been
> > > > > > used
> > > > > > > > as
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > instead of Optional in the code
> example.
> > It
> > > > > > should be
> > > > > > > > > > fine
> > > > > > > > > > > to
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > pseudo code for now and update it after
> > you
> > > > > > submit
> > > > > > > > the
> > > > > > > > > > PR.
> > > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used
> > somewhere
> > > in
> > > > > the
> > > > > > > > > > rejected
> > > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > should be persistContext(...) to keep
> it
> > > > > > consistent.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > multi-instance
> > > > > > > > > > > > > scenarios,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > sure if this is the intended behaviour.
> > If
> > > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > used as a cache, it should be invalid,
> > once
> > > > the
> > > > > > > > related
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > removed from the CatalogStore by
> another
> > > > > > instance.
> > > > > > > > Did
> > > > > > > > > I
> > > > > > > > > > > miss
> > > > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng
> Jin
> > <
> > > > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > > > I think this is a good function
> name, I
> > > > have
> > > > > > > > updated
> > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Some common cache features
> should
> > be
> > > > > > > > implemented
> > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > > > alternative
> > > > > 1
> > > > > > > > turns
> > > > > > > > > > out
> > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > appropriate later, I will improve
> this
> > > part
> > > > > of
> > > > > > the
> > > > > > > > > > > content.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As the above discussion moves
> > forward,
> > > > the
> > > > > > > > option 2
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> > > offline,
> > > > > we
> > > > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > > > inconsistency
> > > > > > > > > > issues.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There are some inconsistent
> > > descriptions
> > > > in
> > > > > > the
> > > > > > > > > > > content.
> > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > > > I will do my best to improve the
> > document
> > > > and
> > > > > > > > > > appreciate
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > can you put the unselected option
> in
> > > > > > `Rejected
> > > > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > > > Sure, I have moved it to the
> `Rejected
> > > > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM
> Shammon
> > > FY
> > > > <
> > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I found there are two options in
> > > > `Proposed
> > > > > > > > > Changes`,
> > > > > > > > > > > can
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > put
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > > > Alternatives`? I
> > > > > > > > > think
> > > > > > > > > > > > this
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM
> Jing
> > Ge
> > > > > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I
> am
> > > > still
> > > > > > > > confused
> > > > > > > > > > > after
> > > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > > the thread with following
> > questions:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> > > registerCatalog()
> > > > > and
> > > > > > > > > > > > addCatalog()
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > > > difference based on their names.
> > One
> > > of
> > > > > > them is
> > > > > > > > > > > > > responsible
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > > persistence. How about
> > > > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > > > persistence.
> > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> > > clearly
> > > > in
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > > Some
> > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > features should be implemented,
> > i.e.
> > > > data
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > cache
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog
> > instance
> > > > > > should be
> > > > > > > > > > found
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > > > initialized
> > > > > > or it
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > > > consistency
> > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > taken
> > > > > > > > > > > > > > care
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > > > forward,
> > > > > > the
> > > > > > > > > > option
> > > > > > > > > > > 2
> > > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > > > because,
> > > > > > afaiu,
> > > > > > > > > > > issues
> > > > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> > > solved
> > > > > > yet. Do
> > > > > > > > > you
> > > > > > > > > > > > still
> > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > > > both options and ask for
> > suggestions
> > > > for
> > > > > > both
> > > > > > > > of
> > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP,
> > there
> > > > are
> > > > > > some
> > > > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the content. Would you like to
> > clean
> > > > > them
> > > > > > up?
> > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM
> Feng
> > > > Jin <
> > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thank you for your response,
> and
> > I
> > > > > > completely
> > > > > > > > > > agree
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > > > complicated
> > > > > > the
> > > > > > > > > whole
> > > > > > > > > > > > issue.
> > > > > > > > > > > > > > > First
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > > > we need to consider the
> > persistence
> > > > of
> > > > > > the
> > > > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > > > persistence
> > > > > > for
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to
> > the
> > > > > > Catalog
> > > > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > > > This method can convert a
> Catalog
> > > > > > instance
> > > > > > > > to a
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > > > implementation
> > > > > > > > will
> > > > > > > > > > > throw
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > > > * Returns a map containing
> the
> > > > > > properties
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > > > * @return a map containing
> the
> > > > > > properties
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > > > * @throws
> > > > > > UnsupportedOperationException if
> > > > > > > > > the
> > > > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > > > * the default implementation
> > of
> > > > this
> > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > > > toProperties()
> > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The specific process is as
> > follows:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. If the user has configured
> a
> > > > > > > > CatalogStore,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > will be called when
> registering a
> > > > > Catalog
> > > > > > > > > > instance
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > > catalog).
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > will be converted to a
> > Map<String,
> > > > > > String>
> > > > > > > > > > > properties
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > > > instances
> > > > > > do
> > > > > > > > not
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > > > CatalogStore,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > > > consistency
> > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > > > conflicts
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> > > itself
> > > > > and
> > > > > > > > > provide
> > > > > > > > > > a
> > > > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > implementation for lazy
> loading,
> > > such
> > > > > as
> > > > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > public class
> ConfigurationCatalog
> > > > > > implements
> > > > > > > > > > > Catalog
> > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > >
> ConfigurationCatalog(Map<String,
> > > > > String>
> > > > > > > > > > > properties)
> > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I have added this design to the
> > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > > > Shammon
> > > > > > FY <
> > > > > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > > > functionality
> > > > > > > > > of
> > > > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > different operations of
> > multiple
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > > > implementation
> > > > > in
> > > > > > > > Trino
> > > > > > > > > > is
> > > > > > > > > > > > also
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino
> only
> > > has
> > > > > one
> > > > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > > > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > > > another
> > > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > save a catalog in
> `Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you
> > mentioned
> > > > > > before,
> > > > > > > > the
> > > > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > > > `CatalogManager`.
> > > > > > > > How
> > > > > > > > > > > about
> > > > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog>
> catalogs`
> > > and
> > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > > > when it is registered or
> added
> > in
> > > > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 5:22 PM
> > > Feng
> > > > > > Jin <
> > > > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I
> > am
> > > > very
> > > > > > sorry
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> > > original
> > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > > pre-discussion
> > > > > > [1]
> > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such
> a
> > > > > > discussion
> > > > > > > > > > before.
> > > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found
> that
> > > the
> > > > > > logic of
> > > > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We
> > cannot
> > > > > only
> > > > > > > > > persist
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > > > not be enough to save
> > Catalogs,
> > > > > > because
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > end
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > > > the configurations
> > > corresponding
> > > > to
> > > > > > the
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > > > CatalogStore
> > > > > > > > > > interface
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > > > part, I
> > > > > > also
> > > > > > > > > took
> > > > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> > handling
> > > > > > catalog
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the
> > suggestion. I
> > > > > will
> > > > > > > > > provide
> > > > > > > > > > a
> > > > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > code examples for this
> part,
> > > and
> > > > > add
> > > > > > it
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > > `addCatalog`
> > > > > > > > > still
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > > catalog
> > > > > to
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> very
> > > > > personal
> > > > > > > > > > opinion,
> > > > > > > > > > > > you
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> > > intuitive
> > > > > to
> > > > > > me.
> > > > > > > > I
> > > > > > > > > > > don't
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> > > statement,
> > > > > > which
> > > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > `addCatalog`?
> > > > > > I'm
> > > > > > > > > > > wondering
> > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my
> > original
> > > > > > design
> > > > > > > > was
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this
> would
> > > > > disrupt
> > > > > > the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > > > configuration to
> > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> > > same
> > > > > time
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really
> neccessary
> > to
> > > > > > provide a
> > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > > > otherwise
> > > > > we
> > > > > > > > would
> > > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > > > Map to store the
> > configuration
> > > > for
> > > > > > lazy
> > > > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > > registration
> > > > > > for
> > > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > > > interface
> > > > > > > > because
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > initialization
> > > > > > > > > for
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > > > inaccurate. It
> > > > > > > > > > > should
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > > > registration. I
> > > > > > > > > have
> > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> > 4:27 PM
> > > > > > Shammon
> > > > > > > > FY <
> > > > > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > > > `registerCatalog`
> > > > > > > > > > > method
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> > > method
> > > > > > named
> > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > > > memory
> > > > > > > > catalog
> > > > > > > > > > is a
> > > > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > > > exposed
> > > > > to
> > > > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > > pre-discussion
> > > > > > [1]
> > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> > > conclusion
> > > > > of
> > > > > > that
> > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > > > interface
> > > > > > and
> > > > > > > > > > > implement
> > > > > > > > > > > > it
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and
> > external
> > > > > > system.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> > > between
> > > > > the
> > > > > > > > > current
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal
> of
> > > the
> > > > > > > > discussion
> > > > > > > > > > in
> > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> > > 3:41 PM
> > > > > > yuxia <
> > > > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> > > this
> > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> > > handling
> > > > > > catalog
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> > > it'll
> > > > be
> > > > > > > > better
> > > > > > > > > to
> > > > > > > > > > > > > explain
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > these methods and give
> a
> > > > brief
> > > > > > > > example
> > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > > `addCatalog`
> > > > > > > > > > still
> > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > > > this method is for
> > storing
> > > > > > catalog to
> > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> > > very
> > > > > > personal
> > > > > > > > > > > > opinion,
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > > > statement,
> > > > > > > > which
> > > > > > > > > > > method
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > > > `addCatalog`?
> > > > > > > > I'm
> > > > > > > > > > > > > wondering
> > > > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with
> SQL
> > > > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really
> > neccessary
> > > to
> > > > > > > > provide a
> > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > > > registration
> > > > > > > > for
> > > > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a
> catalog
> > > > with
> > > > > > CREATE
> > > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> > used
> > > > by
> > > > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> > > registration
> > > > is
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > > > synchronous
> > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> > > `addCatalog`
> > > > > and
> > > > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > > > registration,
> > > > > > > > it
> > > > > > > > > > may
> > > > > > > > > > > > well
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully,
> > but
> > > > then
> > > > > > the
> > > > > > > > > > > following
> > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog
> fail
> > > to
> > > > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break
> > change
> > > > > which
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility
> part.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying
> > asynchronous
> > > > > > > > > > registration, I
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to
> > open
> > > > or
> > > > > > > > register
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance
> > will
> > > be
> > > > > > > > > initialized
> > > > > > > > > > > if
> > > > > > > > > > > > it
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the
> > initialization
> > > > > > process
> > > > > > > > > fails,
> > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > > > initialization
> > > > > > > > > > for
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > > > dev@flink.apache.org
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月
> 03日
> > 下午
> > > > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP
> > 295:
> > > > > > Support
> > > > > > > > > > > > persistence
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you
> > all
> > > > for
> > > > > > your
> > > > > > > > > > > interest
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a
> catalog
> > > > with
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > name
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a
> > crucial
> > > > > > point.
> > > > > > > > > Based
> > > > > > > > > > on
> > > > > > > > > > > > my
> > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated
> catalogs
> > > and
> > > > > > have
> > > > > > > > the
> > > > > > > > > > > > highest
> > > > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> > > that
> > > > > can
> > > > > > > > affect
> > > > > > > > > > the
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1.
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> > > catalog
> > > > > > instance
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2.
> > unregisterCatalog(String
> > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > > catalog
> > > > > > > > > instance
> > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> > > Catalog>
> > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method first
> > retrieves
> > > > the
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String,
> Catalog>
> > > > > > catalogs. If
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the
> > corresponding
> > > > > > > > > configuration
> > > > > > > > > > > > from
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and
> puts
> > > the
> > > > > > > > > initialized
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The following two
> methods
> > > > only
> > > > > > modify
> > > > > > > > > the
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > > > properties
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > > > catalogName
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > > > specified
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The following are
> > possible
> > > > > > conflict
> > > > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the
> corresponding
> > > > > > catalogName
> > > > > > > > > > > already
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in
> > the
> > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> > > saved
> > > > > to
> > > > > > the
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the
> corresponding
> > > > > > catalogName
> > > > > > > > > > > already
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > > > Map<String,
> > > > > > > > > > Catalog>,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> > > catalog)
> > > > > > method
> > > > > > > > > will
> > > > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the
> corresponding
> > > > > > catalogName
> > > > > > > > > > > already
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>,
> the
> > > > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties)
> > method
> > > > can
> > > > > > > > directly
> > > > > > > > > > > save
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > > > getCatalog(String
> > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties
> > for
> > > > > > > > > initialization
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance
> already
> > > > exists
> > > > > > in
> > > > > > > > > > catalogs
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > > > corresponding to
> > > > > > > > > the
> > > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will
> confuse
> > > > users
> > > > > > that
> > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> > > catalog)`
> > > > > in
> > > > > > the
> > > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially
> > lead
> > > > to
> > > > > > > > > > confusion. I
> > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > > > previously
> > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > > > add
> > > `registerCatalog(String
> > > > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)`
> method
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is
> > already
> > > an
> > > > > > > > instance,
> > > > > > > > > > > > adding
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > > > method may not
> > necessarily
> > > > > > result in
> > > > > > > > > lazy
> > > > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think
> > about
> > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > > > necessary,
> > > > > > but
> > > > > > > > > > > perhaps
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented
> in
> > > the
> > > > > > > > > CatalogStore
> > > > > > > > > > > > when
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance,
> the
> > > > > > > > > FileCatalogStore
> > > > > > > > > > > > could
> > > > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> > > while
> > > > > the
> > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need
> the
> > > > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary
> > to
> > > > > have a
> > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which
> > saves
> > > > > > Catalog
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want
> to
> > > > > > implement
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > > > additional
> > > > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > > > registration
> > > > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > > > concern.
> > > > > > > > When
> > > > > > > > > > > > > executing
> > > > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > > > statements that
> > reference a
> > > > > > specific
> > > > > > > > > > > Catalog,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if
> it
> > > has
> > > > > not
> > > > > > > > been
> > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> > > fails,
> > > > > > these
> > > > > > > > > > > > statements
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all
> > have
> > > > the
> > > > > > same
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > > instances.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be
> > resolved
> > > by
> > > > > > > > referring
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that
> the
> > > > > Catalog
> > > > > > > > that
> > > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority
> > for
> > > > > > subsequent
> > > > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > > > 10:58 PM
> > > > > > > > Xianxun
> > > > > > > > > > Ye
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving
> this
> > > > Flip,
> > > > > > I do
> > > > > > > > > > > believe
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I
> > work
> > > > for
> > > > > > also
> > > > > > > > > > > manages a
> > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks
> > becomes
> > > > > slow
> > > > > > > > > because
> > > > > > > > > > of
> > > > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs
> in
> > > the
> > > > > > user's
> > > > > > > > > > Flink
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> > > catalogs
> > > > > to
> > > > > > > > > improve
> > > > > > > > > > > the
> > > > > > > > > > > > > > speed
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > > > After reading
> flip-295, I
> > > > have
> > > > > > some
> > > > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> > > catalog
> > > > > with
> > > > > > > > > CREATE
> > > > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> > used
> > > > by
> > > > > > > > default.
> > > > > > > > > > > What
> > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> > > implement
> > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the
> same
> > > > > > question
> > > > > > > > with
> > > > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > > catalogs
> > > > > and
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have
> > the
> > > > > same
> > > > > > > > > catalog
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > > instances.
> > > > > For
> > > > > > > > > example:
> > > > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1
> ->
> > > > > > hiveCatalog
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > > > define
> > > > > > > > > `catalogs`
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日
> 11:13，Hang
> > > > Ruan
> > > > > <
> > > > > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Samrat Deb <de...@gmail.com>.
Hi Feng,
Thank you for providing the proposal. I believe this feature will be highly
valuable.
I have a couple of inquiries:
1. According to the documentation [1], the createCatalog method does not
mention any exceptions being thrown. However, I would like to confirm if it
is always true that there will be no failures in all scenarios. Please let
me know if there is any additional information I may have missed.
2. Regarding the registration process using the `table.catalog-store.kind`
configuration, could you please provide an exhaustive list of the supported
kinds?
It would be great to have a comprehensive understanding of the options
available.
Bests,
Samrat
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
On Sat, Jun 3, 2023 at 5:54 PM Feng Jin <ji...@gmail.com> wrote:
> Hi Hang and Jark
>
> Thank you very much for your reply.
>
> @Jark
> > 1. Could you move the CatalogStore registration API to the "Public
> Interface" section?
>
> Indeed, it is more reasonable.
>
> > 2. We should prefix "table." for the CatalogStore configuration.
>
> Sure, the names that you provided are indeed more logical and easier to
> understand.
>
> > 3. About the open/close method in CatalogStoreFactory.
>
> I also agree with the proposed requirement scenario and design. I have
> already made modifications to the interface.
>
> @Hang
>
> > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> the resource.
>
> I have already added the missing method.
>
> > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>
> Sorry for the misspelling, I updated this part.
>
> > 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
> to init or close its resource
>
> This is indeed a scenario that needs to be considered, and I have added
> relevant methods accordingly.
> Additionally, I have included some relevant description information in the
> documentation to help others understand
> why the open/close methods need to be added.
>
>
> Best,
> Feng
>
> On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > Thank you for the update.
> >
> > 1. Could you move the CatalogStore registration API to the "Public
> > Interface" section?
> > "Proposed Changes" is more like a place to describe the implementation
> > details.
> >
> > 2. We should prefix "table." for the CatalogStore configuration. Besides,
> > the config key
> > name should be hierarchical[1]. Therefore, it may be better to use:
> > "table.catalog-store.kind"
> > "table.catalog-store.file.path"
> >
> > 3. I think Hang's suggestions make sense.
> >
> > Others look good to me.
> >
> > Best,
> > Jark
> >
> > On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
> >
> > > Hi, Feng.
> > >
> > > Thanks for the update.
> > > The current design looks good to me. I have some minor comments.
> > >
> > > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> > the
> > > resource. For example, when storing the information in MySQL, the store
> > > needs to open and close the connections.
> > >
> > > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> > >
> > > 3. About the usage in the sql gateway.
> > > Considering the usage in sql gateway, the sql gateway may create a
> > > CatalogStore for each session.
> > > If we are using the MySqlCatalogStore, there would be so many
> > connections.
> > > How can we reuse the connection among these sessions?
> > > I think sql gateway need to maintain a connection pool in
> > > the CatalogStoreFactory and each session get its own connection from
> the
> > > pool when it is created.
> > > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> > init
> > > or close its resource.
> > > Or is there a better way?
> > >
> > > Best,
> > > Hang
> > >
> > > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> > >
> > > > Thanks Jingsong.
> > > >
> > > > > Just naming, maybe `createCatalog` in TableEnv
> > > >
> > > > +1 For this, I have already updated FLIP.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > >
> > > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Feng,
> > > > >
> > > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > > functions are converted to createxxx from registerxxx.
> > > > >
> > > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi jark, thanks for your suggestion.
> > > > > >
> > > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > > CatalogStore should be immutable once TableEnv is created.
> > Otherwise,
> > > > > there
> > > > > > might be some data inconsistencies when CatalogStore is changed.
> > > > > >
> > > > > > Yes, We should initialize the CatalogStore when creating the
> > > TableEnv.
> > > > > > Therefore, my current proposal is to add a method to configure
> the
> > > > > > CatalogStore in EnvironmentSettings.
> > > > > >
> > > > > > // Initialize a catalog Store
> > > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > > >
> > > > > > // set up the Table API
> > > > > > final EnvironmentSettings settings =
> > > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > > .withCatalogStore(catalogStore)
> > > > > > .build();
> > > > > >
> > > > > > final TableEnvironment tableEnv =
> > > > > TableEnvironment.create(settings);
> > > > > >
> > > > > >
> > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> default
> > > > > method,
> > > > > > not an interface method?
> > > > > >
> > > > > > Sorry, While I did refer to the Catalog interface, I agree that
> as
> > a
> > > > new
> > > > > > interface, the CatalogStoreFactory should not have a default
> method
> > > but
> > > > > an
> > > > > > interface method. I have already modified the interface.
> > > > > >
> > > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > > deprecated`registerCatalog`.
> > > > > > > 4. In the "Compatibility" section, would be better to mention
> the
> > > > > changed
> > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> other
> > > > > > persisted catalog store) is used.
> > > > > >
> > > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > > >
> > > > > >
> > > > > > [1].
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > > The current design looks pretty good to me. I just have some
> > minor
> > > > > > > comments.
> > > > > > >
> > > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > CatalogStore
> > > > > > > should be immutable once TableEnv is created. Otherwise, there
> > > might
> > > > be
> > > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > > >
> > > > > > > 2. Why does the CatalogStoreFactory interface only have a
> default
> > > > > method,
> > > > > > > not an interface method?
> > > > > > >
> > > > > > > 3. Please mention the alternative API in Javadoc for the
> > deprecated
> > > > > > > `registerCatalog`.
> > > > > > >
> > > > > > > 4. In the "Compatibility" section, would be better to mention
> the
> > > > > changed
> > > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> > other
> > > > > > > persisted catalog store) is used.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Jark
> > > > > > >
> > > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > > >
> > > > > > > > @Ron
> > > > > > > >
> > > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore
> method,
> > > do
> > > > we
> > > > > > > need
> > > > > > > > to provide a default implementation?
> > > > > > > >
> > > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > > > create
> > > > > an
> > > > > > > > InMemoryCatalogStore.
> > > > > > > >
> > > > > > > > > If we get a Catalog from CatalogStore, after initializing
> > it,
> > > > > whether
> > > > > > > we
> > > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > > >
> > > > > > > > Yes, in the current design, catalogs are stored as snapshots,
> > and
> > > > > once
> > > > > > > > initialized, the Catalog will be placed in the Map<String,
> > > Catalog>
> > > > > > > > catalogs.
> > > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > > primary
> > > > > > > source
> > > > > > > > for obtaining the corresponding Catalog.
> > > > > > > >
> > > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > > `catalog.store.path`?
> > > > > > > >
> > > > > > > > I think it is okay. Adding "sql" at the beginning may seem a
> > bit
> > > > > > > strange. I
> > > > > > > > will update the FLIP.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > @Shammon
> > > > > > > >
> > > > > > > > Thank you for the review. I have made the necessary
> > corrections.
> > > > > > > > Regarding the modifications made to the Public Interface, I
> > have
> > > > also
> > > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <
> zjureel@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi feng,
> > > > > > > > >
> > > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > > >
> > > > > > > > > 1. The modification of `CatalogManager` should not be in
> > > `Public
> > > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > > >
> > > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > > > `CatalogStoreFactory`
> > > > > > > > >
> > > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> > should
> > > be
> > > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > > catalogStore.get(catalogName);`
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shammon FY
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <
> ron9.liu@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Feng
> > > > > > > > > >
> > > > > > > > > > Thanks for driving this FLIP, this proposal is very
> useful
> > > for
> > > > > > > catalog
> > > > > > > > > > management.
> > > > > > > > > > I have some small questions:
> > > > > > > > > >
> > > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > > method,
> > > > > do we
> > > > > > > > > need
> > > > > > > > > > to provide a default implementation?
> > > > > > > > > > 2. If we get Catalog from CatalogStore, after
> initializing
> > > it,
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > > > >
> > > > > > > > > > > Hi yuxia
> > > > > > > > > > >
> > > > > > > > > > > > But from the code in Proposed Changes, once we
> > register
> > > > the
> > > > > > > > Catalog,
> > > > > > > > > > we
> > > > > > > > > > > initialize it and open it. right?
> > > > > > > > > > >
> > > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > > original
> > > > > > > CREATE
> > > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > > registerCatalog
> > > > > > > > so
> > > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > > >
> > > > > > > > > > > In the current design, lazy initialization is mainly
> > > > reflected
> > > > > in
> > > > > > > > > > > getCatalog. If CatalogStore has already saved some
> > catalog
> > > > > > > > > > configurations,
> > > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > > luoyuxia@alumni.sjtu.edu.cn
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > > initialization*.
> > > > > > > > If
> > > > > > > > > > i'm
> > > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > > >
> > > > > > > > > > > > IIUC, lazy initialization means only you need to
> access
> > > the
> > > > > > > > catalog,
> > > > > > > > > > then
> > > > > > > > > > > > you initialize it. But from the code in Proposed
> > Changes,
> > > > > once we
> > > > > > > > > > > register
> > > > > > > > > > > > the Catalog,
> > > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Yuxia
> > > > > > > > > > > >
> > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > > Catalog
> > > > > > > > > > configuration
> > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > One of the major changes is that current design will
> > > > provide
> > > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead
> of a
> > > > > cache,
> > > > > > > > which
> > > > > > > > > > > means
> > > > > > > > > > > > once it has been initialized, any changes done by
> other
> > > > > sessions
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > affect it. Point 6 described follow-up options for
> > > further
> > > > > > > > > improvement.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > > jinfeng1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all, I would like to update you on the latest
> > > progress
> > > > > of
> > > > > > > the
> > > > > > > > > > FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon
> FY,
> > > > > ShengKai
> > > > > > > > Fang
> > > > > > > > > > > and I
> > > > > > > > > > > > > had an offline discussion regarding the overall
> > > solution
> > > > > for
> > > > > > > > Flink
> > > > > > > > > > > > > CatalogStore. We have reached a consensus and I
> have
> > > > > updated
> > > > > > > the
> > > > > > > > > > final
> > > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > > configuration
> > > > > > > > > similar
> > > > > > > > > > > to
> > > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > > 2.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > > storeCatalog(String
> > > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > > CatalogDescriptor
> > > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > > will both use CatalogDescriptor instead of
> Catalog
> > > > > instance.
> > > > > > > > > This
> > > > > > > > > > > way,
> > > > > > > > > > > > > CatalogStore will only be responsible for saving
> > and
> > > > > > > > retrieving
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > configurations without having to initialize
> > > catalogs.
> > > > > > > > > > > > > 3.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The default registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > > > function in CatalogManager will be marked as
> > > > deprecated.
> > > > > > > > > > > > > 4.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A new function registerCatalog(String
> catalogName,
> > > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > > catalog) will be added to serve as the default
> > > > > registration
> > > > > > > > > > function
> > > > > > > > > > > > for
> > > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > > 5.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager
> will
> > > > > remain
> > > > > > > > > > unchanged
> > > > > > > > > > > > and
> > > > > > > > > > > > > save initialized catalogs.This means that
> deletion
> > > > > > > operations
> > > > > > > > > from
> > > > > > > > > > > one
> > > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > > 6.
> > > > > > > > > > > > >
> > > > > > > > > > > > > To support multi-session synchronization
> scenarios
> > > for
> > > > > > > > deletions
> > > > > > > > > > > later
> > > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > > configurable.There
> > > > > > > > > > may
> > > > > > > > > > > > be
> > > > > > > > > > > > > three possible situations:
> > > > > > > > > > > > >
> > > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > > >
> > > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > > automatically
> > > > > > > > clear
> > > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > c.No caching of any instances; each call to
> > > getCatalog
> > > > > > > > creates a
> > > > > > > > > > new
> > > > > > > > > > > > > instance.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you very much for your attention and
> > suggestions
> > > on
> > > > > this
> > > > > > > > > > FLIP. A
> > > > > > > > > > > > > special thanks to Hang Ruan for his suggestions on
> > the
> > > > > entire
> > > > > > > > > design
> > > > > > > > > > > and
> > > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you have any further suggestions or feedback
> about
> > > > this
> > > > > FLIP
> > > > > > > > > > please
> > > > > > > > > > > > feel
> > > > > > > > > > > > > free to share.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to
> me.
> > > We
> > > > > could
> > > > > > > > > still
> > > > > > > > > > > > > > reconsider in the future how to provide more
> common
> > > > > built-in
> > > > > > > > > cache
> > > > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > > > CatalogSotre
> > > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There might be more such issues. I would
> > suggest
> > > > you
> > > > > > > > > > completely
> > > > > > > > > > > > walk
> > > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I am very sorry for my carelessness and at the
> > same
> > > > > time, I
> > > > > > > > > > greatly
> > > > > > > > > > > > > > > appreciate your careful review. I have
> thoroughly
> > > > > checked
> > > > > > > the
> > > > > > > > > > > entire
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > > > design,
> > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > could work without Optional CatalogStore being
> > > > > configured.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, in the original design, CatalogStore was
> not
> > > > > necessary
> > > > > > > > > > because
> > > > > > > > > > > > > > > CatalogManager used Map<String, Catalog>
> catalogs
> > > to
> > > > > store
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > > Therefore, I
> > > > > > > > > modified
> > > > > > > > > > > this
> > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > > > catalogs
> > > > > > > from
> > > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve
> as
> > > the
> > > > > > > default
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > > functionality of
> > > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > > The previous plan that kept
> > > > > Map<String,Catalog>catalogs has
> > > > > > > > > been
> > > > > > > > > > > > moved
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are still many places contain
> > inconsistent
> > > > > content,
> > > > > > > > > e.g.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > > > 2. The java comment of the method
> > > > > registerCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> > tell
> > > > > what the
> > > > > > > > > > method
> > > > > > > > > > > is
> > > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There might be more such issues. I would
> > suggest
> > > > you
> > > > > > > > > completely
> > > > > > > > > > > > walk
> > > > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> > that
> > > > you
> > > > > will
> > > > > > > > > build
> > > > > > > > > > > the
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > functionality in the CatalogStore? This is a
> > very
> > > > > > > different
> > > > > > > > > > > design
> > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > from what the current FLIP described. If I am
> > not
> > > > > > > mistaken,
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > current FLIP design, CatalogManager could
> work
> > > > > without
> > > > > > > > > Optional
> > > > > > > > > > > > > > > > CatalogStore being configured. That is the
> > reason
> > > > > why I
> > > > > > > > > > mentioned
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > last email that the example code wrt the
> > Optional
> > > > > > > > > CatalogStore
> > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are still some NIT issues in the
> FLIP
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you very much for the careful
> review. I
> > > > have
> > > > > > > > already
> > > > > > > > > > made
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > > multi-instance
> > > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > > > scenarios
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > design. I am thinking whether we should
> > remove
> > > > > > > > 'Map<String,
> > > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > > InMemoryCatalogStore.
> > > > > > > > > This
> > > > > > > > > > > > way,
> > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > can implement their own CatalogStore to
> > achieve
> > > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > > non-conflicting scenarios. What do you
> think?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> > > still
> > > > > some
> > > > > > > > NIT
> > > > > > > > > > > issues
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore
> has
> > > been
> > > > > used
> > > > > > > as
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > instead of Optional in the code example.
> It
> > > > > should be
> > > > > > > > > fine
> > > > > > > > > > to
> > > > > > > > > > > > use
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > pseudo code for now and update it after
> you
> > > > > submit
> > > > > > > the
> > > > > > > > > PR.
> > > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used
> somewhere
> > in
> > > > the
> > > > > > > > > rejected
> > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > > > consistent.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > multi-instance
> > > > > > > > > > > > scenarios,
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > sure if this is the intended behaviour.
> If
> > > > > > > Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > used as a cache, it should be invalid,
> once
> > > the
> > > > > > > related
> > > > > > > > > > > catalog
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > > > instance.
> > > > > > > Did
> > > > > > > > I
> > > > > > > > > > miss
> > > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin
> <
> > > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > > I think this is a good function name, I
> > > have
> > > > > > > updated
> > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Some common cache features should
> be
> > > > > > > implemented
> > > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > > alternative
> > > > 1
> > > > > > > turns
> > > > > > > > > out
> > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > appropriate later, I will improve this
> > part
> > > > of
> > > > > the
> > > > > > > > > > content.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As the above discussion moves
> forward,
> > > the
> > > > > > > option 2
> > > > > > > > > > > > solution
> > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> > offline,
> > > > we
> > > > > > > think
> > > > > > > > > that
> > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > > inconsistency
> > > > > > > > > issues.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There are some inconsistent
> > descriptions
> > > in
> > > > > the
> > > > > > > > > > content.
> > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > > I will do my best to improve the
> document
> > > and
> > > > > > > > > appreciate
> > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > > > `Rejected
> > > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon
> > FY
> > > <
> > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I found there are two options in
> > > `Proposed
> > > > > > > > Changes`,
> > > > > > > > > > can
> > > > > > > > > > > > you
> > > > > > > > > > > > > > put
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > > Alternatives`? I
> > > > > > > > think
> > > > > > > > > > > this
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing
> Ge
> > > > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> > > still
> > > > > > > confused
> > > > > > > > > > after
> > > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > > the thread with following
> questions:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> > registerCatalog()
> > > > and
> > > > > > > > > > > addCatalog()
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > > difference based on their names.
> One
> > of
> > > > > them is
> > > > > > > > > > > > responsible
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > persistence. How about
> > > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > > 2. As you mentioned that
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs
> > > > > > > > > > > is
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > > persistence.
> > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> > clearly
> > > in
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > > > > Some
> > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > features should be implemented,
> i.e.
> > > data
> > > > > in
> > > > > > > the
> > > > > > > > > > cache
> > > > > > > > > > > > and
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog
> instance
> > > > > should be
> > > > > > > > > found
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > > initialized
> > > > > or it
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > > consistency
> > > > > will
> > > > > > > > be
> > > > > > > > > > > taken
> > > > > > > > > > > > > care
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > > forward,
> > > > > the
> > > > > > > > > option
> > > > > > > > > > 2
> > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > > because,
> > > > > afaiu,
> > > > > > > > > > issues
> > > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> > solved
> > > > > yet. Do
> > > > > > > > you
> > > > > > > > > > > still
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > > both options and ask for
> suggestions
> > > for
> > > > > both
> > > > > > > of
> > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP,
> there
> > > are
> > > > > some
> > > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the content. Would you like to
> clean
> > > > them
> > > > > up?
> > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> > > Jin <
> > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you for your response, and
> I
> > > > > completely
> > > > > > > > > agree
> > > > > > > > > > > > with
> > > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > > complicated
> > > > > the
> > > > > > > > whole
> > > > > > > > > > > issue.
> > > > > > > > > > > > > > First
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > > we need to consider the
> persistence
> > > of
> > > > > the
> > > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > > persistence
> > > > > for
> > > > > > > > > Catalog
> > > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to
> the
> > > > > Catalog
> > > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > > > instance
> > > > > > > to a
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > > implementation
> > > > > > > will
> > > > > > > > > > throw
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > > > properties
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > > > properties
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > > * @throws
> > > > > UnsupportedOperationException if
> > > > > > > > the
> > > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > > * the default implementation
> of
> > > this
> > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > > toProperties()
> > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The specific process is as
> follows:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > > > CatalogStore,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > will be called when registering a
> > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > > catalog).
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > will be converted to a
> Map<String,
> > > > > String>
> > > > > > > > > > properties
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > > instances
> > > > > do
> > > > > > > not
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > > CatalogStore,
> > > > > > > > the
> > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > > consistency
> > > > > with
> > > > > > > > the
> > > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > > conflicts
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> > itself
> > > > and
> > > > > > > > provide
> > > > > > > > > a
> > > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > implementation for lazy loading,
> > such
> > > > as
> > > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > > > implements
> > > > > > > > > > Catalog
> > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > > > String>
> > > > > > > > > > properties)
> > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have added this design to the
> > FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > > Shammon
> > > > > FY <
> > > > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > > functionality
> > > > > > > > of
> > > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > different operations of
> multiple
> > > > > > > > > > `registerCatalog`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > > implementation
> > > > in
> > > > > > > Trino
> > > > > > > > > is
> > > > > > > > > > > also
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only
> > has
> > > > one
> > > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > > another
> > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > > > Catalog>
> > > > > > > > > catalogs`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you
> mentioned
> > > > > before,
> > > > > > > the
> > > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > > `CatalogManager`.
> > > > > > > How
> > > > > > > > > > about
> > > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs`
> > and
> > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > > when it is registered or added
> in
> > > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM
> > Feng
> > > > > Jin <
> > > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I
> am
> > > very
> > > > > sorry
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> > original
> > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > pre-discussion
> > > > > [1]
> > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > > > discussion
> > > > > > > > > before.
> > > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > whole solution, I found that
> > the
> > > > > logic of
> > > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > > need to change much. *We
> cannot
> > > > only
> > > > > > > > persist
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > > registerCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > > not be enough to save
> Catalogs,
> > > > > because
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > end
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > > the configurations
> > corresponding
> > > to
> > > > > the
> > > > > > > > > Catalog
> > > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > > CatalogStore
> > > > > > > > > interface
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > > part, I
> > > > > also
> > > > > > > > took
> > > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> handling
> > > > > catalog
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > > Thank you for the
> suggestion. I
> > > > will
> > > > > > > > provide
> > > > > > > > > a
> > > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > code examples for this part,
> > and
> > > > add
> > > > > it
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > `addCatalog`
> > > > > > > > still
> > > > > > > > > > > looks
> > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > catalog
> > > > to
> > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > > personal
> > > > > > > > > opinion,
> > > > > > > > > > > you
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> > intuitive
> > > > to
> > > > > me.
> > > > > > > I
> > > > > > > > > > don't
> > > > > > > > > > > > see
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> > statement,
> > > > > which
> > > > > > > > > method
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > `addCatalog`?
> > > > > I'm
> > > > > > > > > > wondering
> > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my
> original
> > > > > design
> > > > > > > was
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > > > disrupt
> > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > > configuration to
> > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> > same
> > > > time
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary
> to
> > > > > provide a
> > > > > > > > > > default
> > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > interface
> > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > > otherwise
> > > > we
> > > > > > > would
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > > Map to store the
> configuration
> > > for
> > > > > lazy
> > > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > registration
> > > > > for
> > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > > interface
> > > > > > > because
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > initialization
> > > > > > > > for
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > > inaccurate. It
> > > > > > > > > > should
> > > > > > > > > > > be
> > > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > > registration. I
> > > > > > > > have
> > > > > > > > > > > > already
> > > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 4:27 PM
> > > > > Shammon
> > > > > > > FY <
> > > > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > > `registerCatalog`
> > > > > > > > > > method
> > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> > method
> > > > > named
> > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > > memory
> > > > > > > catalog
> > > > > > > > > is a
> > > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > > exposed
> > > > to
> > > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > > pre-discussion
> > > > > [1]
> > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> > conclusion
> > > > of
> > > > > that
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > > interface
> > > > > and
> > > > > > > > > > implement
> > > > > > > > > > > it
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > > as memory, file and
> external
> > > > > system.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> > between
> > > > the
> > > > > > > > current
> > > > > > > > > > FLIP
> > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > > What about the proposal of
> > the
> > > > > > > discussion
> > > > > > > > > in
> > > > > > > > > > > > thread
> > > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> > 3:41 PM
> > > > > yuxia <
> > > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> > this
> > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> > handling
> > > > > catalog
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> > it'll
> > > be
> > > > > > > better
> > > > > > > > to
> > > > > > > > > > > > explain
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> > > brief
> > > > > > > example
> > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > > `addCatalog`
> > > > > > > > > still
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > > this method is for
> storing
> > > > > catalog to
> > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> > very
> > > > > personal
> > > > > > > > > > > opinion,
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > > statement,
> > > > > > > which
> > > > > > > > > > method
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > > `addCatalog`?
> > > > > > > I'm
> > > > > > > > > > > > wondering
> > > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really
> neccessary
> > to
> > > > > > > provide a
> > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > > registration
> > > > > > > for
> > > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> > > with
> > > > > CREATE
> > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> used
> > > by
> > > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> > registration
> > > is
> > > > > the
> > > > > > > > > default
> > > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > > synchronous
> > > > > > > > > > > > registration
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> > `addCatalog`
> > > > and
> > > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > > registration,
> > > > > > > it
> > > > > > > > > may
> > > > > > > > > > > well
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > > executes successfully,
> but
> > > then
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail
> > to
> > > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break
> change
> > > > which
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying
> asynchronous
> > > > > > > > > registration, I
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to
> open
> > > or
> > > > > > > register
> > > > > > > > > > > catalog
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance
> will
> > be
> > > > > > > > initialized
> > > > > > > > > > if
> > > > > > > > > > > it
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > > yet. If the
> initialization
> > > > > process
> > > > > > > > fails,
> > > > > > > > > > > these
> > > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > > initialization
> > > > > > > > > for
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > > dev@flink.apache.org
> > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日
> 下午
> > > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP
> 295:
> > > > > Support
> > > > > > > > > > > persistence
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> > registration
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you
> all
> > > for
> > > > > your
> > > > > > > > > > interest
> > > > > > > > > > > in
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> > > with
> > > > > the
> > > > > > > same
> > > > > > > > > > name
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a
> crucial
> > > > > point.
> > > > > > > > Based
> > > > > > > > > on
> > > > > > > > > > > my
> > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs
> > and
> > > > > have
> > > > > > > the
> > > > > > > > > > > highest
> > > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> > that
> > > > can
> > > > > > > affect
> > > > > > > > > the
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> > catalog
> > > > > instance
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2.
> unregisterCatalog(String
> > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > catalog
> > > > > > > > instance
> > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method first
> retrieves
> > > the
> > > > > > > > > > corresponding
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > > > catalogs. If
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > retrieves the
> corresponding
> > > > > > > > configuration
> > > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts
> > the
> > > > > > > > initialized
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> > > only
> > > > > modify
> > > > > > > > the
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > > properties
> > > > > to
> > > > > > > the
> > > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > > catalogName
> > > > > with
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > > specified
> > > > > > > > > > > configuration
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The following are
> possible
> > > > > conflict
> > > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > > > catalogName
> > > > > > > > > > already
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in
> the
> > > > > > > Map<String,
> > > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> > saved
> > > > to
> > > > > the
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > > > catalogName
> > > > > > > > > > already
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > > Map<String,
> > > > > > > > > Catalog>,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> > catalog)
> > > > > method
> > > > > > > > will
> > > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > > > catalogName
> > > > > > > > > > already
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > String> properties)
> method
> > > can
> > > > > > > directly
> > > > > > > > > > save
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > > getCatalog(String
> > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > > use the new properties
> for
> > > > > > > > initialization
> > > > > > > > > > > > because
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> > > exists
> > > > > in
> > > > > > > > > catalogs
> > > > > > > > > > > and
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > > corresponding to
> > > > > > > > the
> > > > > > > > > > > > original
> > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> > > users
> > > > > that
> > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> > catalog)`
> > > > in
> > > > > the
> > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > catalogName,
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This could potentially
> lead
> > > to
> > > > > > > > > confusion. I
> > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > > previously
> > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > > add
> > `registerCatalog(String
> > > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is
> already
> > an
> > > > > > > instance,
> > > > > > > > > > > adding
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > > registerCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > > method may not
> necessarily
> > > > > result in
> > > > > > > > lazy
> > > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think
> about
> > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > > necessary,
> > > > > but
> > > > > > > > > > perhaps
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > > should be implemented in
> > the
> > > > > > > > CatalogStore
> > > > > > > > > > > when
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > > > FileCatalogStore
> > > > > > > > > > > could
> > > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> > while
> > > > the
> > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary
> to
> > > > have a
> > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > > implementation, which
> saves
> > > > > Catalog
> > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > > > implement
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > > additional
> > > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > > registration
> > > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > > concern.
> > > > > > > When
> > > > > > > > > > > > executing
> > > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > > statements that
> reference a
> > > > > specific
> > > > > > > > > > Catalog,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it
> > has
> > > > not
> > > > > > > been
> > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> > fails,
> > > > > these
> > > > > > > > > > > statements
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > > Catalog>
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all
> have
> > > the
> > > > > same
> > > > > > > > > > catalog
> > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > instances.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This issue can be
> resolved
> > by
> > > > > > > referring
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > > > Catalog
> > > > > > > that
> > > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > > will be given priority
> for
> > > > > subsequent
> > > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > > 10:58 PM
> > > > > > > Xianxun
> > > > > > > > > Ye
> > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> > > Flip,
> > > > > I do
> > > > > > > > > > believe
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I
> work
> > > for
> > > > > also
> > > > > > > > > > manages a
> > > > > > > > > > > > lot
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks
> becomes
> > > > slow
> > > > > > > > because
> > > > > > > > > of
> > > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in
> > the
> > > > > user's
> > > > > > > > > Flink
> > > > > > > > > > > SQL
> > > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> > catalogs
> > > > to
> > > > > > > > improve
> > > > > > > > > > the
> > > > > > > > > > > > > speed
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> > > have
> > > > > some
> > > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> > catalog
> > > > with
> > > > > > > > CREATE
> > > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > > registration method is
> used
> > > by
> > > > > > > default.
> > > > > > > > > > What
> > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> > implement
> > > > > > > > asynchronous
> > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > > > question
> > > > > > > with
> > > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > catalogs
> > > > and
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have
> the
> > > > same
> > > > > > > > catalog
> > > > > > > > > > > name,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> > instances.
> > > > For
> > > > > > > > example:
> > > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > > > hiveCatalog
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > > define
> > > > > > > > `catalogs`
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> > > Ruan
> > > > <
> > > > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi Hang and Jark
Thank you very much for your reply.
@Jark
> 1. Could you move the CatalogStore registration API to the "Public
Interface" section?
Indeed, it is more reasonable.
> 2. We should prefix "table." for the CatalogStore configuration.
Sure, the names that you provided are indeed more logical and easier to
understand.
> 3. About the open/close method in CatalogStoreFactory.
I also agree with the proposed requirement scenario and design. I have
already made modifications to the interface.
@Hang
> 1. The `CatalogStore` need the `open`/`close` methods to init or close
the resource.
I have already added the missing method.
> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
Sorry for the misspelling, I updated this part.
> 3. Then the `CatalogStoreFactory` may need the `open`/`close` methods
to init or close its resource
This is indeed a scenario that needs to be considered, and I have added
relevant methods accordingly.
Additionally, I have included some relevant description information in the
documentation to help others understand
why the open/close methods need to be added.
Best,
Feng
On Sat, Jun 3, 2023 at 12:35 PM Jark Wu <im...@gmail.com> wrote:
> Hi Jing,
>
> Thank you for the update.
>
> 1. Could you move the CatalogStore registration API to the "Public
> Interface" section?
> "Proposed Changes" is more like a place to describe the implementation
> details.
>
> 2. We should prefix "table." for the CatalogStore configuration. Besides,
> the config key
> name should be hierarchical[1]. Therefore, it may be better to use:
> "table.catalog-store.kind"
> "table.catalog-store.file.path"
>
> 3. I think Hang's suggestions make sense.
>
> Others look good to me.
>
> Best,
> Jark
>
> On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
>
> > Hi, Feng.
> >
> > Thanks for the update.
> > The current design looks good to me. I have some minor comments.
> >
> > 1. The `CatalogStore` need the `open`/`close` methods to init or close
> the
> > resource. For example, when storing the information in MySQL, the store
> > needs to open and close the connections.
> >
> > 2. The `getCatalog` is misspelled as `optionalDescriptor`.
> >
> > 3. About the usage in the sql gateway.
> > Considering the usage in sql gateway, the sql gateway may create a
> > CatalogStore for each session.
> > If we are using the MySqlCatalogStore, there would be so many
> connections.
> > How can we reuse the connection among these sessions?
> > I think sql gateway need to maintain a connection pool in
> > the CatalogStoreFactory and each session get its own connection from the
> > pool when it is created.
> > Then the `CatalogStoreFactory` may need the `open`/`close` methods to
> init
> > or close its resource.
> > Or is there a better way?
> >
> > Best,
> > Hang
> >
> > Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> >
> > > Thanks Jingsong.
> > >
> > > > Just naming, maybe `createCatalog` in TableEnv
> > >
> > > +1 For this, I have already updated FLIP.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Feng,
> > > >
> > > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > > functions are converted to createxxx from registerxxx.
> > > >
> > > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi jark, thanks for your suggestion.
> > > > >
> > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > > CatalogStore should be immutable once TableEnv is created.
> Otherwise,
> > > > there
> > > > > might be some data inconsistencies when CatalogStore is changed.
> > > > >
> > > > > Yes, We should initialize the CatalogStore when creating the
> > TableEnv.
> > > > > Therefore, my current proposal is to add a method to configure the
> > > > > CatalogStore in EnvironmentSettings.
> > > > >
> > > > > // Initialize a catalog Store
> > > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > > >
> > > > > // set up the Table API
> > > > > final EnvironmentSettings settings =
> > > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > > .withCatalogStore(catalogStore)
> > > > > .build();
> > > > >
> > > > > final TableEnvironment tableEnv =
> > > > TableEnvironment.create(settings);
> > > > >
> > > > >
> > > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > > method,
> > > > > not an interface method?
> > > > >
> > > > > Sorry, While I did refer to the Catalog interface, I agree that as
> a
> > > new
> > > > > interface, the CatalogStoreFactory should not have a default method
> > but
> > > > an
> > > > > interface method. I have already modified the interface.
> > > > >
> > > > > > 3. Please mention the alternative API in Javadoc for the
> > > > > deprecated`registerCatalog`.
> > > > > > 4. In the "Compatibility" section, would be better to mention the
> > > > changed
> > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > > persisted catalog store) is used.
> > > > >
> > > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > > >
> > > > >
> > > > > [1].
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > > The current design looks pretty good to me. I just have some
> minor
> > > > > > comments.
> > > > > >
> > > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > CatalogStore
> > > > > > should be immutable once TableEnv is created. Otherwise, there
> > might
> > > be
> > > > > > some data inconsistencies when CatalogStore is changed.
> > > > > >
> > > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > > method,
> > > > > > not an interface method?
> > > > > >
> > > > > > 3. Please mention the alternative API in Javadoc for the
> deprecated
> > > > > > `registerCatalog`.
> > > > > >
> > > > > > 4. In the "Compatibility" section, would be better to mention the
> > > > changed
> > > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or
> other
> > > > > > persisted catalog store) is used.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi ， thanks all for reviewing the flip.
> > > > > > >
> > > > > > > @Ron
> > > > > > >
> > > > > > > > Regarding the CatalogStoreFactory#createCatalogStore method,
> > do
> > > we
> > > > > > need
> > > > > > > to provide a default implementation?
> > > > > > >
> > > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > > create
> > > > an
> > > > > > > InMemoryCatalogStore.
> > > > > > >
> > > > > > > > If we get a Catalog from CatalogStore, after initializing
> it,
> > > > whether
> > > > > > we
> > > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > > >
> > > > > > > Yes, in the current design, catalogs are stored as snapshots,
> and
> > > > once
> > > > > > > initialized, the Catalog will be placed in the Map<String,
> > Catalog>
> > > > > > > catalogs.
> > > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> > primary
> > > > > > source
> > > > > > > for obtaining the corresponding Catalog.
> > > > > > >
> > > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > > `catalog.store.path`?
> > > > > > >
> > > > > > > I think it is okay. Adding "sql" at the beginning may seem a
> bit
> > > > > > strange. I
> > > > > > > will update the FLIP.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > @Shammon
> > > > > > >
> > > > > > > Thank you for the review. I have made the necessary
> corrections.
> > > > > > > Regarding the modifications made to the Public Interface, I
> have
> > > also
> > > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi feng,
> > > > > > > >
> > > > > > > > Thanks for updating, I have some minor comments
> > > > > > > >
> > > > > > > > 1. The modification of `CatalogManager` should not be in
> > `Public
> > > > > > > > Interfaces`, it is not a public interface.
> > > > > > > >
> > > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > > `CatalogStoreFactory`
> > > > > > > >
> > > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > > catalogStore.get(catalogName);` in the `CatalogManager`
> should
> > be
> > > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > > catalogStore.get(catalogName);`
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shammon FY
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Feng
> > > > > > > > >
> > > > > > > > > Thanks for driving this FLIP, this proposal is very useful
> > for
> > > > > > catalog
> > > > > > > > > management.
> > > > > > > > > I have some small questions:
> > > > > > > > >
> > > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> > method,
> > > > do we
> > > > > > > > need
> > > > > > > > > to provide a default implementation?
> > > > > > > > > 2. If we get Catalog from CatalogStore, after initializing
> > it,
> > > > > > whether
> > > > > > > we
> > > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > > >
> > > > > > > > > > Hi yuxia
> > > > > > > > > >
> > > > > > > > > > > But from the code in Proposed Changes, once we
> register
> > > the
> > > > > > > Catalog,
> > > > > > > > > we
> > > > > > > > > > initialize it and open it. right?
> > > > > > > > > >
> > > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> > original
> > > > > > CREATE
> > > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > > registerCatalog
> > > > > > > so
> > > > > > > > > > that parameter validation can be performed.
> > > > > > > > > >
> > > > > > > > > > In the current design, lazy initialization is mainly
> > > reflected
> > > > in
> > > > > > > > > > getCatalog. If CatalogStore has already saved some
> catalog
> > > > > > > > > configurations,
> > > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > > luoyuxia@alumni.sjtu.edu.cn
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Feng.
> > > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > > initialization*.
> > > > > > > If
> > > > > > > > > i'm
> > > > > > > > > > > wrong, please correct me.
> > > > > > > > > > >
> > > > > > > > > > > IIUC, lazy initialization means only you need to access
> > the
> > > > > > > catalog,
> > > > > > > > > then
> > > > > > > > > > > you initialize it. But from the code in Proposed
> Changes,
> > > > once we
> > > > > > > > > > register
> > > > > > > > > > > the Catalog,
> > > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Yuxia
> > > > > > > > > > >
> > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> > Catalog
> > > > > > > > > configuration
> > > > > > > > > > > and asynchronous registration
> > > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > > >
> > > > > > > > > > > One of the major changes is that current design will
> > > provide
> > > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > > > cache,
> > > > > > > which
> > > > > > > > > > means
> > > > > > > > > > > once it has been initialized, any changes done by other
> > > > sessions
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > affect it. Point 6 described follow-up options for
> > further
> > > > > > > > improvement.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > > jinfeng1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all, I would like to update you on the latest
> > progress
> > > > of
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > > > ShengKai
> > > > > > > Fang
> > > > > > > > > > and I
> > > > > > > > > > > > had an offline discussion regarding the overall
> > solution
> > > > for
> > > > > > > Flink
> > > > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > > > updated
> > > > > > the
> > > > > > > > > final
> > > > > > > > > > > > solution in FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > >
> > > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > > configuration
> > > > > > > > similar
> > > > > > > > > > to
> > > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > > 2.
> > > > > > > > > > > >
> > > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > > storeCatalog(String
> > > > > > > > > > > > catalogName, CatalogDescriptor) and
> > CatalogDescriptor
> > > > > > > > > > > getCatalog(String)
> > > > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > > > instance.
> > > > > > > > This
> > > > > > > > > > way,
> > > > > > > > > > > > CatalogStore will only be responsible for saving
> and
> > > > > > > retrieving
> > > > > > > > > > > catalog
> > > > > > > > > > > > configurations without having to initialize
> > catalogs.
> > > > > > > > > > > > 3.
> > > > > > > > > > > >
> > > > > > > > > > > > The default registerCatalog(String catalogName,
> > > Catalog
> > > > > > > catalog)
> > > > > > > > > > > > function in CatalogManager will be marked as
> > > deprecated.
> > > > > > > > > > > > 4.
> > > > > > > > > > > >
> > > > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > > > CatalogDescriptor
> > > > > > > > > > > > catalog) will be added to serve as the default
> > > > registration
> > > > > > > > > function
> > > > > > > > > > > for
> > > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > > 5.
> > > > > > > > > > > >
> > > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > > > remain
> > > > > > > > > unchanged
> > > > > > > > > > > and
> > > > > > > > > > > > save initialized catalogs.This means that deletion
> > > > > > operations
> > > > > > > > from
> > > > > > > > > > one
> > > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > > 6.
> > > > > > > > > > > >
> > > > > > > > > > > > To support multi-session synchronization scenarios
> > for
> > > > > > > deletions
> > > > > > > > > > later
> > > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > > configurable.There
> > > > > > > > > may
> > > > > > > > > > > be
> > > > > > > > > > > > three possible situations:
> > > > > > > > > > > >
> > > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > > >
> > > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > > automatically
> > > > > > > clear
> > > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > > >
> > > > > > > > > > > > c.No caching of any instances; each call to
> > getCatalog
> > > > > > > creates a
> > > > > > > > > new
> > > > > > > > > > > > instance.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > > >
> > > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you very much for your attention and
> suggestions
> > on
> > > > this
> > > > > > > > > FLIP. A
> > > > > > > > > > > > special thanks to Hang Ruan for his suggestions on
> the
> > > > entire
> > > > > > > > design
> > > > > > > > > > and
> > > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > > >
> > > > > > > > > > > > If you have any further suggestions or feedback about
> > > this
> > > > FLIP
> > > > > > > > > please
> > > > > > > > > > > feel
> > > > > > > > > > > > free to share.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for improving the FLIP. It looks good to me.
> > We
> > > > could
> > > > > > > > still
> > > > > > > > > > > > > reconsider in the future how to provide more common
> > > > built-in
> > > > > > > > cache
> > > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > > CatalogSotre
> > > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There might be more such issues. I would
> suggest
> > > you
> > > > > > > > > completely
> > > > > > > > > > > walk
> > > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am very sorry for my carelessness and at the
> same
> > > > time, I
> > > > > > > > > greatly
> > > > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > > > checked
> > > > > > the
> > > > > > > > > > entire
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > > design,
> > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > could work without Optional CatalogStore being
> > > > configured.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > > > necessary
> > > > > > > > > because
> > > > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs
> > to
> > > > store
> > > > > > > > > catalog
> > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > However, this caused inconsistency issues.
> > > Therefore, I
> > > > > > > > modified
> > > > > > > > > > this
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > > catalogs
> > > > > > from
> > > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > > At the same time, InMemoryCatalog will serve as
> > the
> > > > > > default
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > to save catalogs in memory and replace the
> > > > functionality of
> > > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > > The previous plan that kept
> > > > Map<String,Catalog>catalogs has
> > > > > > > > been
> > > > > > > > > > > moved
> > > > > > > > > > > > to
> > > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are still many places contain
> inconsistent
> > > > content,
> > > > > > > > e.g.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > > 2. The java comment of the method
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not
> tell
> > > > what the
> > > > > > > > > method
> > > > > > > > > > is
> > > > > > > > > > > > > > doing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There might be more such issues. I would
> suggest
> > > you
> > > > > > > > completely
> > > > > > > > > > > walk
> > > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean
> that
> > > you
> > > > will
> > > > > > > > build
> > > > > > > > > > the
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > functionality in the CatalogStore? This is a
> very
> > > > > > different
> > > > > > > > > > design
> > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > from what the current FLIP described. If I am
> not
> > > > > > mistaken,
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > > > without
> > > > > > > > Optional
> > > > > > > > > > > > > > > CatalogStore being configured. That is the
> reason
> > > > why I
> > > > > > > > > mentioned
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > last email that the example code wrt the
> Optional
> > > > > > > > CatalogStore
> > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you very much for the careful review. I
> > > have
> > > > > > > already
> > > > > > > > > made
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > > multi-instance
> > > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > > scenarios
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > current
> > > > > > > > > > > > > > > > design. I am thinking whether we should
> remove
> > > > > > > 'Map<String,
> > > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > > InMemoryCatalogStore.
> > > > > > > > This
> > > > > > > > > > > way,
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > can implement their own CatalogStore to
> achieve
> > > > > > > > > multi-instance
> > > > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> > still
> > > > some
> > > > > > > NIT
> > > > > > > > > > issues
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has
> > been
> > > > used
> > > > > > as
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > instead of Optional in the code example. It
> > > > should be
> > > > > > > > fine
> > > > > > > > > to
> > > > > > > > > > > use
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > pseudo code for now and update it after you
> > > > submit
> > > > > > the
> > > > > > > > PR.
> > > > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere
> in
> > > the
> > > > > > > > rejected
> > > > > > > > > > > > section
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > > consistent.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > multi-instance
> > > > > > > > > > > scenarios,
> > > > > > > > > > > > I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > used as a cache, it should be invalid, once
> > the
> > > > > > related
> > > > > > > > > > catalog
> > > > > > > > > > > > has
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > > instance.
> > > > > > Did
> > > > > > > I
> > > > > > > > > miss
> > > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > > I think this is a good function name, I
> > have
> > > > > > updated
> > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Some common cache features should be
> > > > > > implemented
> > > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> > alternative
> > > 1
> > > > > > turns
> > > > > > > > out
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > appropriate later, I will improve this
> part
> > > of
> > > > the
> > > > > > > > > content.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As the above discussion moves forward,
> > the
> > > > > > option 2
> > > > > > > > > > > solution
> > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > > Yes, after discussing with Shammon
> offline,
> > > we
> > > > > > think
> > > > > > > > that
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > > inconsistency
> > > > > > > > issues.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are some inconsistent
> descriptions
> > in
> > > > the
> > > > > > > > > content.
> > > > > > > > > > > > Would
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > > I will do my best to improve the document
> > and
> > > > > > > > appreciate
> > > > > > > > > > your
> > > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > > `Rejected
> > > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > > Alternatives`.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon
> FY
> > <
> > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I found there are two options in
> > `Proposed
> > > > > > > Changes`,
> > > > > > > > > can
> > > > > > > > > > > you
> > > > > > > > > > > > > put
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > > Alternatives`? I
> > > > > > > think
> > > > > > > > > > this
> > > > > > > > > > > > may
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> > still
> > > > > > confused
> > > > > > > > > after
> > > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Naming confusion -
> registerCatalog()
> > > and
> > > > > > > > > > addCatalog()
> > > > > > > > > > > > have
> > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > > difference based on their names. One
> of
> > > > them is
> > > > > > > > > > > responsible
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > persistence. How about
> > persistCatalog()?
> > > > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > is
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > > persistence.
> > > > > > I
> > > > > > > > > would
> > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > > their purpose conceptually and
> clearly
> > in
> > > > the
> > > > > > > FLIP.
> > > > > > > > > > Some
> > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > features should be implemented, i.e.
> > data
> > > > in
> > > > > > the
> > > > > > > > > cache
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > > > should be
> > > > > > > > found
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the cache(either it has been
> > initialized
> > > > or it
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > > consistency
> > > > will
> > > > > > > be
> > > > > > > > > > taken
> > > > > > > > > > > > care
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> > forward,
> > > > the
> > > > > > > > option
> > > > > > > > > 2
> > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> > because,
> > > > afaiu,
> > > > > > > > > issues
> > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > previously with option 1 are not
> solved
> > > > yet. Do
> > > > > > > you
> > > > > > > > > > still
> > > > > > > > > > > > > want
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > > both options and ask for suggestions
> > for
> > > > both
> > > > > > of
> > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there
> > are
> > > > some
> > > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the content. Would you like to clean
> > > them
> > > > up?
> > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> > Jin <
> > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > > > completely
> > > > > > > > agree
> > > > > > > > > > > with
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > > Initially, I may have over
> > complicated
> > > > the
> > > > > > > whole
> > > > > > > > > > issue.
> > > > > > > > > > > > > First
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > > we need to consider the persistence
> > of
> > > > the
> > > > > > > > > Catalog's
> > > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > > If we only need to provide
> > persistence
> > > > for
> > > > > > > > Catalog
> > > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > > > Catalog
> > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > > instance
> > > > > > to a
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties, and the default
> > > > implementation
> > > > > > will
> > > > > > > > > throw
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > > properties
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > > properties
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > > * @throws
> > > > UnsupportedOperationException if
> > > > > > > the
> > > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > > * the default implementation of
> > this
> > > > > > method
> > > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > > toProperties()
> > > > > > {
> > > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > > CatalogStore,
> > > > > > > > the
> > > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will be called when registering a
> > > Catalog
> > > > > > > > instance
> > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > > catalog).
> > > > > > > > > > > The
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > > > String>
> > > > > > > > > properties
> > > > > > > > > > > and
> > > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> > instances
> > > > do
> > > > > > not
> > > > > > > > > > > implement
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > > CatalogStore,
> > > > > > > the
> > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > > consistency
> > > > with
> > > > > > > the
> > > > > > > > > > > > original
> > > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > the same time can also avoid
> > conflicts
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > > we can start from the Catalog
> itself
> > > and
> > > > > > > provide
> > > > > > > > a
> > > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > implementation for lazy loading,
> such
> > > as
> > > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > > implements
> > > > > > > > > Catalog
> > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > > String>
> > > > > > > > > properties)
> > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have added this design to the
> FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> > Shammon
> > > > FY <
> > > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > > functionality
> > > > > > > of
> > > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > > > `registerCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> > implementation
> > > in
> > > > > > Trino
> > > > > > > > is
> > > > > > > > > > also
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only
> has
> > > one
> > > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalog to memory and
> > `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> > another
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > > Catalog>
> > > > > > > > catalogs`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > > > before,
> > > > > > the
> > > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > > `CatalogManager`.
> > > > > > How
> > > > > > > > > about
> > > > > > > > > > > > > saving
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs`
> and
> > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM
> Feng
> > > > Jin <
> > > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am
> > very
> > > > sorry
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > > by my deviation from the
> original
> > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > pre-discussion
> > > > [1]
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > > discussion
> > > > > > > > before.
> > > > > > > > > > > > > However,
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > whole solution, I found that
> the
> > > > logic of
> > > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> > > only
> > > > > > > persist
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > > > because
> > > > > > in
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > > we
> > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > > the configurations
> corresponding
> > to
> > > > the
> > > > > > > > Catalog
> > > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > > CatalogStore
> > > > > > > > interface
> > > > > > > > > > for
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> > part, I
> > > > also
> > > > > > > took
> > > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > > catalog
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> > > will
> > > > > > > provide
> > > > > > > > a
> > > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > code examples for this part,
> and
> > > add
> > > > it
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > `addCatalog`
> > > > > > > still
> > > > > > > > > > looks
> > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > catalog
> > > to
> > > > > > > > > > CatalogStore,
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > personal
> > > > > > > > opinion,
> > > > > > > > > > you
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more
> intuitive
> > > to
> > > > me.
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > see
> > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG
> statement,
> > > > which
> > > > > > > > method
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > `addCatalog`?
> > > > I'm
> > > > > > > > > wondering
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > stement.
> > > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > > > design
> > > > > > was
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > it
> > > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > > disrupt
> > > > the
> > > > > > > > > existing
> > > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > > configuration to
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the
> same
> > > time
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > > > provide a
> > > > > > > > > default
> > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > interface
> `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> > otherwise
> > > we
> > > > > > would
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > > Map to store the configuration
> > for
> > > > lazy
> > > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > registration
> > > > for
> > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > I don't think
> > > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> > interface
> > > > > > because
> > > > > > > > > > Catalog
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > initialization
> > > > > > > for
> > > > > > > > > > > catalog
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > > inaccurate. It
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > > registration. I
> > > > > > > have
> > > > > > > > > > > already
> > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > > > Shammon
> > > > > > FY <
> > > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > > `registerCatalog`
> > > > > > > > > method
> > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > confuse users whether a
> method
> > > > named
> > > > > > > > > > `addCatalog`
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> > memory
> > > > > > catalog
> > > > > > > > is a
> > > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> > exposed
> > > to
> > > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I found there is a
> > pre-discussion
> > > > [1]
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the
> conclusion
> > > of
> > > > that
> > > > > > > > > > > discussion
> > > > > > > > > > > > is
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> > interface
> > > > and
> > > > > > > > > implement
> > > > > > > > > > it
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > > > system.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think there is a gap
> between
> > > the
> > > > > > > current
> > > > > > > > > FLIP
> > > > > > > > > > > > > design
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > > What about the proposal of
> the
> > > > > > discussion
> > > > > > > > in
> > > > > > > > > > > thread
> > > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > >
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at
> 3:41 PM
> > > > yuxia <
> > > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving
> this
> > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of
> handling
> > > > catalog
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think
> it'll
> > be
> > > > > > better
> > > > > > > to
> > > > > > > > > > > explain
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> > brief
> > > > > > example
> > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > > `addCatalog`
> > > > > > > > still
> > > > > > > > > > > looks
> > > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > > catalog to
> > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's
> very
> > > > personal
> > > > > > > > > > opinion,
> > > > > > > > > > > > you
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > > statement,
> > > > > > which
> > > > > > > > > method
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > > `addCatalog`?
> > > > > > I'm
> > > > > > > > > > > wondering
> > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > > stement.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary
> to
> > > > > > provide a
> > > > > > > > > > default
> > > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > interface
> > > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > > registration
> > > > > > for
> > > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> > with
> > > > CREATE
> > > > > > > > > > CATALOG,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> > by
> > > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > > If asynchronous
> registration
> > is
> > > > the
> > > > > > > > default
> > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > > synchronous
> > > > > > > > > > > registration
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > > Will both method
> `addCatalog`
> > > and
> > > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > > registration,
> > > > > > it
> > > > > > > > may
> > > > > > > > > > well
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > > executes successfully, but
> > then
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail
> to
> > > > open.
> > > > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> > > which
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > > > registration, I
> > > > > > > > > > > would
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > > will be an executor to open
> > or
> > > > > > register
> > > > > > > > > > catalog
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will
> be
> > > > > > > initialized
> > > > > > > > > if
> > > > > > > > > > it
> > > > > > > > > > > > has
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > > > process
> > > > > > > fails,
> > > > > > > > > > these
> > > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > > initialization
> > > > > > > > for
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > > jinfeng1995@gmail.com
> > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > > dev@flink.apache.org
> > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > > > Support
> > > > > > > > > > persistence
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > and asynchronous
> registration
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all
> > for
> > > > your
> > > > > > > > > interest
> > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> > with
> > > > the
> > > > > > same
> > > > > > > > > name
> > > > > > > > > > > that
> > > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > > > point.
> > > > > > > Based
> > > > > > > > on
> > > > > > > > > > my
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs
> and
> > > > have
> > > > > > the
> > > > > > > > > > highest
> > > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > There are three methods
> that
> > > can
> > > > > > affect
> > > > > > > > the
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method puts the
> catalog
> > > > instance
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > catalog
> > > > > > > instance
> > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > > from the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method first retrieves
> > the
> > > > > > > > > corresponding
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > > catalogs. If
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > > > configuration
> > > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts
> the
> > > > > > > initialized
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> > only
> > > > modify
> > > > > > > the
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > > properties
> > > > to
> > > > > > the
> > > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> > catalogName
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > > specified
> > > > > > > > > > configuration
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > > > conflict
> > > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > > > Map<String,
> > > > > > > > > > > Catalog>,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly
> saved
> > > to
> > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > > Map<String,
> > > > > > > > Catalog>,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog
> catalog)
> > > > method
> > > > > > > will
> > > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > > catalogName
> > > > > > > > > already
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > > addCatalog(String
> > > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > String> properties) method
> > can
> > > > > > directly
> > > > > > > > > save
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > > getCatalog(String
> > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > > > initialization
> > > > > > > > > > > because
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> > exists
> > > > in
> > > > > > > > catalogs
> > > > > > > > > > and
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > > corresponding to
> > > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> > users
> > > > that
> > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog
> catalog)`
> > > in
> > > > the
> > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > > catalogName,
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This could potentially lead
> > to
> > > > > > > > confusion. I
> > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > > addCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > > previously
> > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > > add
> `registerCatalog(String
> > > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already
> an
> > > > > > instance,
> > > > > > > > > > adding
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > > registerCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > > > result in
> > > > > > > lazy
> > > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > > necessary,
> > > > but
> > > > > > > > > perhaps
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > > should be implemented in
> the
> > > > > > > CatalogStore
> > > > > > > > > > when
> > > > > > > > > > > > the
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > > FileCatalogStore
> > > > > > > > > > could
> > > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file,
> while
> > > the
> > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> > > have a
> > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > > > Catalog
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > > implement
> > > > > > > > > > asynchronous
> > > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > > additional
> > > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > > registration
> > > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > > concern.
> > > > > > When
> > > > > > > > > > > executing
> > > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > > > specific
> > > > > > > > > Catalog,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it
> has
> > > not
> > > > > > been
> > > > > > > > > > > > initialized
> > > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > initialization process
> fails,
> > > > these
> > > > > > > > > > statements
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> > Catalog>
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have
> > the
> > > > same
> > > > > > > > > catalog
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> instances.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved
> by
> > > > > > referring
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > > Catalog
> > > > > > that
> > > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > > > subsequent
> > > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> > 10:58 PM
> > > > > > Xianxun
> > > > > > > > Ye
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> > Flip,
> > > > I do
> > > > > > > > > believe
> > > > > > > > > > > this
> > > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work
> > for
> > > > also
> > > > > > > > > manages a
> > > > > > > > > > > lot
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> > > slow
> > > > > > > because
> > > > > > > > of
> > > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in
> the
> > > > user's
> > > > > > > > Flink
> > > > > > > > > > SQL
> > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all
> catalogs
> > > to
> > > > > > > improve
> > > > > > > > > the
> > > > > > > > > > > > speed
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> > have
> > > > some
> > > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a
> catalog
> > > with
> > > > > > > CREATE
> > > > > > > > > > > CATALOG,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> > by
> > > > > > default.
> > > > > > > > > What
> > > > > > > > > > if
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > > failed? And how to
> implement
> > > > > > > asynchronous
> > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > > question
> > > > > > with
> > > > > > > > > > > Shammon’s
> > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> catalogs
> > > and
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> > > same
> > > > > > > catalog
> > > > > > > > > > name,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > different catalog
> instances.
> > > For
> > > > > > > example:
> > > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > > hiveCatalog
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> > define
> > > > > > > `catalogs`
> > > > > > > > > and
> > > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> > Ruan
> > > <
> > > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jark Wu <im...@gmail.com>.
Hi Jing,
Thank you for the update.
1. Could you move the CatalogStore registration API to the "Public
Interface" section?
"Proposed Changes" is more like a place to describe the implementation
details.
2. We should prefix "table." for the CatalogStore configuration. Besides,
the config key
name should be hierarchical[1]. Therefore, it may be better to use:
"table.catalog-store.kind"
"table.catalog-store.file.path"
3. I think Hang's suggestions make sense.
Others look good to me.
Best,
Jark
On Fri, 2 Jun 2023 at 17:28, Hang Ruan <ru...@gmail.com> wrote:
> Hi, Feng.
>
> Thanks for the update.
> The current design looks good to me. I have some minor comments.
>
> 1. The `CatalogStore` need the `open`/`close` methods to init or close the
> resource. For example, when storing the information in MySQL, the store
> needs to open and close the connections.
>
> 2. The `getCatalog` is misspelled as `optionalDescriptor`.
>
> 3. About the usage in the sql gateway.
> Considering the usage in sql gateway, the sql gateway may create a
> CatalogStore for each session.
> If we are using the MySqlCatalogStore, there would be so many connections.
> How can we reuse the connection among these sessions?
> I think sql gateway need to maintain a connection pool in
> the CatalogStoreFactory and each session get its own connection from the
> pool when it is created.
> Then the `CatalogStoreFactory` may need the `open`/`close` methods to init
> or close its resource.
> Or is there a better way?
>
> Best,
> Hang
>
> Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
>
> > Thanks Jingsong.
> >
> > > Just naming, maybe `createCatalog` in TableEnv
> >
> > +1 For this, I have already updated FLIP.
> >
> >
> > Best,
> > Feng
> >
> >
> > On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> > wrote:
> >
> > > Thanks Feng,
> > >
> > > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > > functions are converted to createxxx from registerxxx.
> > >
> > > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com>
> wrote:
> > > >
> > > > Hi jark, thanks for your suggestion.
> > > >
> > > > > 1. How to register the CatalogStore for Table API? I think the
> > > > CatalogStore should be immutable once TableEnv is created. Otherwise,
> > > there
> > > > might be some data inconsistencies when CatalogStore is changed.
> > > >
> > > > Yes, We should initialize the CatalogStore when creating the
> TableEnv.
> > > > Therefore, my current proposal is to add a method to configure the
> > > > CatalogStore in EnvironmentSettings.
> > > >
> > > > // Initialize a catalog Store
> > > > CatalogStore catalogStore = new FileCatalogStore("");
> > > >
> > > > // set up the Table API
> > > > final EnvironmentSettings settings =
> > > > EnvironmentSettings.newInstance().inBatchMode()
> > > > .withCatalogStore(catalogStore)
> > > > .build();
> > > >
> > > > final TableEnvironment tableEnv =
> > > TableEnvironment.create(settings);
> > > >
> > > >
> > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > method,
> > > > not an interface method?
> > > >
> > > > Sorry, While I did refer to the Catalog interface, I agree that as a
> > new
> > > > interface, the CatalogStoreFactory should not have a default method
> but
> > > an
> > > > interface method. I have already modified the interface.
> > > >
> > > > > 3. Please mention the alternative API in Javadoc for the
> > > > deprecated`registerCatalog`.
> > > > > 4. In the "Compatibility" section, would be better to mention the
> > > changed
> > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > persisted catalog store) is used.
> > > >
> > > > Thanks for the suggestion, I have updated the FLIP [1].
> > > >
> > > >
> > > > [1].
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > > >
> > > > > Hi Feng,
> > > > >
> > > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > > The current design looks pretty good to me. I just have some minor
> > > > > comments.
> > > > >
> > > > > 1. How to register the CatalogStore for Table API? I think the
> > > CatalogStore
> > > > > should be immutable once TableEnv is created. Otherwise, there
> might
> > be
> > > > > some data inconsistencies when CatalogStore is changed.
> > > > >
> > > > > 2. Why does the CatalogStoreFactory interface only have a default
> > > method,
> > > > > not an interface method?
> > > > >
> > > > > 3. Please mention the alternative API in Javadoc for the deprecated
> > > > > `registerCatalog`.
> > > > >
> > > > > 4. In the "Compatibility" section, would be better to mention the
> > > changed
> > > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > > persisted catalog store) is used.
> > > > >
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi ， thanks all for reviewing the flip.
> > > > > >
> > > > > > @Ron
> > > > > >
> > > > > > > Regarding the CatalogStoreFactory#createCatalogStore method,
> do
> > we
> > > > > need
> > > > > > to provide a default implementation?
> > > > > >
> > > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> > create
> > > an
> > > > > > InMemoryCatalogStore.
> > > > > >
> > > > > > > If we get a Catalog from CatalogStore, after initializing it,
> > > whether
> > > > > we
> > > > > > put it in Map<String, Catalog> catalogs again?
> > > > > >
> > > > > > Yes, in the current design, catalogs are stored as snapshots, and
> > > once
> > > > > > initialized, the Catalog will be placed in the Map<String,
> Catalog>
> > > > > > catalogs.
> > > > > > Subsequently, the Map<String, Catalog> catalogs will be the
> primary
> > > > > source
> > > > > > for obtaining the corresponding Catalog.
> > > > > >
> > > > > > > how about renaming them to `catalog.store.type` and
> > > > > > `catalog.store.path`?
> > > > > >
> > > > > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > > > > strange. I
> > > > > > will update the FLIP.
> > > > > >
> > > > > >
> > > > > >
> > > > > > @Shammon
> > > > > >
> > > > > > Thank you for the review. I have made the necessary corrections.
> > > > > > Regarding the modifications made to the Public Interface, I have
> > also
> > > > > > included the relevant changes to the `TableEnvironment`.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > >
> > > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi feng,
> > > > > > >
> > > > > > > Thanks for updating, I have some minor comments
> > > > > > >
> > > > > > > 1. The modification of `CatalogManager` should not be in
> `Public
> > > > > > > Interfaces`, it is not a public interface.
> > > > > > >
> > > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > > `CatalogStoreFactory`
> > > > > > >
> > > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > > catalogStore.get(catalogName);` in the `CatalogManager` should
> be
> > > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > > catalogStore.get(catalogName);`
> > > > > > >
> > > > > > > Best,
> > > > > > > Shammon FY
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi, Feng
> > > > > > > >
> > > > > > > > Thanks for driving this FLIP, this proposal is very useful
> for
> > > > > catalog
> > > > > > > > management.
> > > > > > > > I have some small questions:
> > > > > > > >
> > > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore
> method,
> > > do we
> > > > > > > need
> > > > > > > > to provide a default implementation?
> > > > > > > > 2. If we get Catalog from CatalogStore, after initializing
> it,
> > > > > whether
> > > > > > we
> > > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > > >
> > > > > > > > > Hi yuxia
> > > > > > > > >
> > > > > > > > > > But from the code in Proposed Changes, once we register
> > the
> > > > > > Catalog,
> > > > > > > > we
> > > > > > > > > initialize it and open it. right?
> > > > > > > > >
> > > > > > > > > Yes, In order to avoid inconsistent semantics of the
> original
> > > > > CREATE
> > > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > > registerCatalog
> > > > > > so
> > > > > > > > > that parameter validation can be performed.
> > > > > > > > >
> > > > > > > > > In the current design, lazy initialization is mainly
> > reflected
> > > in
> > > > > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > > > > configurations,
> > > > > > > > > only initialization is required in getCatalog.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > > luoyuxia@alumni.sjtu.edu.cn
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Feng.
> > > > > > > > > > I'm trying to understanding the meaning of *lazy
> > > initialization*.
> > > > > > If
> > > > > > > > i'm
> > > > > > > > > > wrong, please correct me.
> > > > > > > > > >
> > > > > > > > > > IIUC, lazy initialization means only you need to access
> the
> > > > > > catalog,
> > > > > > > > then
> > > > > > > > > > you initialize it. But from the code in Proposed Changes,
> > > once we
> > > > > > > > > register
> > > > > > > > > > the Catalog,
> > > > > > > > > > we initialize it and open it. right?
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Yuxia
> > > > > > > > > >
> > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of
> Catalog
> > > > > > > > configuration
> > > > > > > > > > and asynchronous registration
> > > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > > >
> > > > > > > > > > One of the major changes is that current design will
> > provide
> > > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > > cache,
> > > > > > which
> > > > > > > > > means
> > > > > > > > > > once it has been initialized, any changes done by other
> > > sessions
> > > > > > will
> > > > > > > > not
> > > > > > > > > > affect it. Point 6 described follow-up options for
> further
> > > > > > > improvement.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > > jinfeng1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all, I would like to update you on the latest
> progress
> > > of
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > > ShengKai
> > > > > > Fang
> > > > > > > > > and I
> > > > > > > > > > > had an offline discussion regarding the overall
> solution
> > > for
> > > > > > Flink
> > > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > > updated
> > > > > the
> > > > > > > > final
> > > > > > > > > > > solution in FLIP.
> > > > > > > > > > >
> > > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > >
> > > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > > configuration
> > > > > > > similar
> > > > > > > > > to
> > > > > > > > > > > TableDescriptor.
> > > > > > > > > > > 2.
> > > > > > > > > > >
> > > > > > > > > > > The two key functions of CatalogStore - void
> > > > > > storeCatalog(String
> > > > > > > > > > > catalogName, CatalogDescriptor) and
> CatalogDescriptor
> > > > > > > > > > getCatalog(String)
> > > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > > instance.
> > > > > > > This
> > > > > > > > > way,
> > > > > > > > > > > CatalogStore will only be responsible for saving and
> > > > > > retrieving
> > > > > > > > > > catalog
> > > > > > > > > > > configurations without having to initialize
> catalogs.
> > > > > > > > > > > 3.
> > > > > > > > > > >
> > > > > > > > > > > The default registerCatalog(String catalogName,
> > Catalog
> > > > > > catalog)
> > > > > > > > > > > function in CatalogManager will be marked as
> > deprecated.
> > > > > > > > > > > 4.
> > > > > > > > > > >
> > > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > > CatalogDescriptor
> > > > > > > > > > > catalog) will be added to serve as the default
> > > registration
> > > > > > > > function
> > > > > > > > > > for
> > > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > > 5.
> > > > > > > > > > >
> > > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > > remain
> > > > > > > > unchanged
> > > > > > > > > > and
> > > > > > > > > > > save initialized catalogs.This means that deletion
> > > > > operations
> > > > > > > from
> > > > > > > > > one
> > > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > > 6.
> > > > > > > > > > >
> > > > > > > > > > > To support multi-session synchronization scenarios
> for
> > > > > > deletions
> > > > > > > > > later
> > > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > > configurable.There
> > > > > > > > may
> > > > > > > > > > be
> > > > > > > > > > > three possible situations:
> > > > > > > > > > >
> > > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > > >
> > > > > > > > > > > b.Introduction of LRU cache logic which can
> > > automatically
> > > > > > clear
> > > > > > > > > > > long-unused catalogs.
> > > > > > > > > > >
> > > > > > > > > > > c.No caching of any instances; each call to
> getCatalog
> > > > > > creates a
> > > > > > > > new
> > > > > > > > > > > instance.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This is the document for discussion:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > > >
> > > > > > > > > > > This is the final proposal document:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thank you very much for your attention and suggestions
> on
> > > this
> > > > > > > > FLIP. A
> > > > > > > > > > > special thanks to Hang Ruan for his suggestions on the
> > > entire
> > > > > > > design
> > > > > > > > > and
> > > > > > > > > > > organizing offline discussions.
> > > > > > > > > > >
> > > > > > > > > > > If you have any further suggestions or feedback about
> > this
> > > FLIP
> > > > > > > > please
> > > > > > > > > > feel
> > > > > > > > > > > free to share.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for improving the FLIP. It looks good to me.
> We
> > > could
> > > > > > > still
> > > > > > > > > > > > reconsider in the future how to provide more common
> > > built-in
> > > > > > > cache
> > > > > > > > > > > > functionality in CatalogManager so that not every
> > > > > CatalogSotre
> > > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > > jinfeng1995@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There might be more such issues. I would suggest
> > you
> > > > > > > > completely
> > > > > > > > > > walk
> > > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am very sorry for my carelessness and at the same
> > > time, I
> > > > > > > > greatly
> > > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > > checked
> > > > > the
> > > > > > > > > entire
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> > design,
> > > > > > > > > > CatalogManager
> > > > > > > > > > > > > could work without Optional CatalogStore being
> > > configured.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > > necessary
> > > > > > > > because
> > > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs
> to
> > > store
> > > > > > > > catalog
> > > > > > > > > > > > > instances.
> > > > > > > > > > > > > However, this caused inconsistency issues.
> > Therefore, I
> > > > > > > modified
> > > > > > > > > this
> > > > > > > > > > > > part
> > > > > > > > > > > > > of the design and removed Map<String, Catalog>
> > catalogs
> > > > > from
> > > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > > At the same time, InMemoryCatalog will serve as
> the
> > > > > default
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > to save catalogs in memory and replace the
> > > functionality of
> > > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > > The previous plan that kept
> > > Map<String,Catalog>catalogs has
> > > > > > > been
> > > > > > > > > > moved
> > > > > > > > > > > to
> > > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There are still many places contain inconsistent
> > > content,
> > > > > > > e.g.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > > 2. The java comment of the method
> > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog catalog) in CatalogManager does not tell
> > > what the
> > > > > > > > method
> > > > > > > > > is
> > > > > > > > > > > > > doing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There might be more such issues. I would suggest
> > you
> > > > > > > completely
> > > > > > > > > > walk
> > > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean that
> > you
> > > will
> > > > > > > build
> > > > > > > > > the
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > functionality in the CatalogStore? This is a very
> > > > > different
> > > > > > > > > design
> > > > > > > > > > > > > concept
> > > > > > > > > > > > > > from what the current FLIP described. If I am not
> > > > > mistaken,
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > > without
> > > > > > > Optional
> > > > > > > > > > > > > > CatalogStore being configured. That is the reason
> > > why I
> > > > > > > > mentioned
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > last email that the example code wrt the Optional
> > > > > > > CatalogStore
> > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > > > > > correct.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you very much for the careful review. I
> > have
> > > > > > already
> > > > > > > > made
> > > > > > > > > > the
> > > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > > multi-instance
> > > > > > > > > > > scenarios, I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > > scenarios
> > > > > with
> > > > > > > the
> > > > > > > > > > > current
> > > > > > > > > > > > > > > design. I am thinking whether we should remove
> > > > > > 'Map<String,
> > > > > > > > > > > Catalog>'
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > make Cache the default behavior of
> > > > > InMemoryCatalogStore.
> > > > > > > This
> > > > > > > > > > way,
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > > > > multi-instance
> > > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for working on the FLIP. There are
> still
> > > some
> > > > > > NIT
> > > > > > > > > issues
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has
> been
> > > used
> > > > > as
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > instead of Optional in the code example. It
> > > should be
> > > > > > > fine
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > it
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > pseudo code for now and update it after you
> > > submit
> > > > > the
> > > > > > > PR.
> > > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in
> > the
> > > > > > > rejected
> > > > > > > > > > > section
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > > consistent.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > multi-instance
> > > > > > > > > > scenarios,
> > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > used as a cache, it should be invalid, once
> the
> > > > > related
> > > > > > > > > catalog
> > > > > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > removed from the CatalogStore by another
> > > instance.
> > > > > Did
> > > > > > I
> > > > > > > > miss
> > > > > > > > > > > > > > something?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > > I think this is a good function name, I
> have
> > > > > updated
> > > > > > it
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Some common cache features should be
> > > > > implemented
> > > > > > > > > > > > > > > > > Thank you for the suggestion. If
> alternative
> > 1
> > > > > turns
> > > > > > > out
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > appropriate later, I will improve this part
> > of
> > > the
> > > > > > > > content.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As the above discussion moves forward,
> the
> > > > > option 2
> > > > > > > > > > solution
> > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > > Yes, after discussing with Shammon offline,
> > we
> > > > > think
> > > > > > > that
> > > > > > > > > > > > solution
> > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > be more suitable and also avoid any
> > > inconsistency
> > > > > > > issues.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There are some inconsistent descriptions
> in
> > > the
> > > > > > > > content.
> > > > > > > > > > > Would
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > > I will do my best to improve the document
> and
> > > > > > > appreciate
> > > > > > > > > your
> > > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > can you put the unselected option in
> > > `Rejected
> > > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > > Alternatives`.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY
> <
> > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I found there are two options in
> `Proposed
> > > > > > Changes`,
> > > > > > > > can
> > > > > > > > > > you
> > > > > > > > > > > > put
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > unselected option in `Rejected
> > > Alternatives`? I
> > > > > > think
> > > > > > > > > this
> > > > > > > > > > > may
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am
> still
> > > > > confused
> > > > > > > > after
> > > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog()
> > and
> > > > > > > > > addCatalog()
> > > > > > > > > > > have
> > > > > > > > > > > > > no
> > > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > > difference based on their names. One of
> > > them is
> > > > > > > > > > responsible
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > persistence. How about
> persistCatalog()?
> > > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > is
> > > > > > > > > > > > used
> > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > > persistence.
> > > > > I
> > > > > > > > would
> > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > > their purpose conceptually and clearly
> in
> > > the
> > > > > > FLIP.
> > > > > > > > > Some
> > > > > > > > > > > > common
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > features should be implemented, i.e.
> data
> > > in
> > > > > the
> > > > > > > > cache
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > > should be
> > > > > > > found
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > store
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the cache(either it has been
> initialized
> > > or it
> > > > > > will
> > > > > > > > be
> > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > > for the same catalog name. The
> > consistency
> > > will
> > > > > > be
> > > > > > > > > taken
> > > > > > > > > > > care
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > > 3. As the above discussion moves
> forward,
> > > the
> > > > > > > option
> > > > > > > > 2
> > > > > > > > > > > > solution
> > > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > like a replacement of option 1,
> because,
> > > afaiu,
> > > > > > > > issues
> > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > previously with option 1 are not solved
> > > yet. Do
> > > > > > you
> > > > > > > > > still
> > > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > > both options and ask for suggestions
> for
> > > both
> > > > > of
> > > > > > > > them?
> > > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there
> are
> > > some
> > > > > > > > > > inconsistent
> > > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the content. Would you like to clean
> > them
> > > up?
> > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng
> Jin <
> > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > > completely
> > > > > > > agree
> > > > > > > > > > with
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > > Initially, I may have over
> complicated
> > > the
> > > > > > whole
> > > > > > > > > issue.
> > > > > > > > > > > > First
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > > we need to consider the persistence
> of
> > > the
> > > > > > > > Catalog's
> > > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > > If we only need to provide
> persistence
> > > for
> > > > > > > Catalog
> > > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > > Catalog
> > > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > > instance
> > > > > to a
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > properties, and the default
> > > implementation
> > > > > will
> > > > > > > > throw
> > > > > > > > > > an
> > > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > > properties
> > > > > of
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > > * @return a map containing the
> > > properties
> > > > > of
> > > > > > > the
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > > * @throws
> > > UnsupportedOperationException if
> > > > > > the
> > > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > > * the default implementation of
> this
> > > > > method
> > > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > > default Map<String, String>
> > > toProperties()
> > > > > {
> > > > > > > > > > > > > > > > > > > > throw new
> > > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > > CatalogStore,
> > > > > > > the
> > > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > will be called when registering a
> > Catalog
> > > > > > > instance
> > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > Catalog
> > > > > > > > catalog).
> > > > > > > > > > The
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > > String>
> > > > > > > > properties
> > > > > > > > > > and
> > > > > > > > > > > > > saved
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog
> instances
> > > do
> > > > > not
> > > > > > > > > > implement
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > > CatalogStore,
> > > > > > the
> > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > will not be called, ensuring
> > consistency
> > > with
> > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > the same time can also avoid
> conflicts
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > > we can start from the Catalog itself
> > and
> > > > > > provide
> > > > > > > a
> > > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > implementation for lazy loading, such
> > as
> > > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > > implements
> > > > > > > > Catalog
> > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> > String>
> > > > > > > > properties)
> > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM
> Shammon
> > > FY <
> > > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I have no questions about the
> > > functionality
> > > > > > of
> > > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > > `registerCatalog`
> > > > > > > > > or
> > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > `CatalogManager`. The
> implementation
> > in
> > > > > Trino
> > > > > > > is
> > > > > > > > > also
> > > > > > > > > > > the
> > > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has
> > one
> > > > > > > > > > `createCatalog`,
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalog to memory and
> `CatalogStore`.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think we don't need to add
> another
> > > > > > > > > > `registerCatalog`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> > Catalog>
> > > > > > > catalogs`
> > > > > > > > > or
> > > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > > before,
> > > > > the
> > > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > > `CatalogManager`.
> > > > > How
> > > > > > > > about
> > > > > > > > > > > > saving
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > > > > `CatalogStore
> > > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng
> > > Jin <
> > > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am
> very
> > > sorry
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > I found there is a
> pre-discussion
> > > [1]
> > > > > for
> > > > > > > > this
> > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > > discussion
> > > > > > > before.
> > > > > > > > > > > > However,
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > whole solution, I found that the
> > > logic of
> > > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> > only
> > > > > > persist
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > > so exposing only
> > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > > because
> > > > > in
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > we
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > > the configurations corresponding
> to
> > > the
> > > > > > > Catalog
> > > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > > decided to introduce the
> > CatalogStore
> > > > > > > interface
> > > > > > > > > for
> > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > persistence. Regarding this
> part, I
> > > also
> > > > > > took
> > > > > > > > > > > > inspiration
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > catalog
> > > > > with
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > name
> > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> > will
> > > > > > provide
> > > > > > > a
> > > > > > > > > > > detailed
> > > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > code examples for this part, and
> > add
> > > it
> > > > > to
> > > > > > > the
> > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > `addCatalog`
> > > > > > still
> > > > > > > > > looks
> > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > this method is for storing
> catalog
> > to
> > > > > > > > > CatalogStore,
> > > > > > > > > > > how
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > personal
> > > > > > > opinion,
> > > > > > > > > you
> > > > > > > > > > > can
> > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive
> > to
> > > me.
> > > > > I
> > > > > > > > don't
> > > > > > > > > > see
> > > > > > > > > > > > any
> > > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement,
> > > which
> > > > > > > method
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> `addCatalog`?
> > > I'm
> > > > > > > > wondering
> > > > > > > > > > > > whether
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> stement.
> > > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > > design
> > > > > was
> > > > > > to
> > > > > > > > add
> > > > > > > > > > it
> > > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> > disrupt
> > > the
> > > > > > > > existing
> > > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > > configuration to
> > > > > > the
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > > the Catalog instance at the same
> > time
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > > provide a
> > > > > > > > default
> > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > I think it is necessary,
> otherwise
> > we
> > > > > would
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > > Map to store the configuration
> for
> > > lazy
> > > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > registration
> > > for
> > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > I don't think
> > registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > made into an asynchronous
> interface
> > > > > because
> > > > > > > > > Catalog
> > > > > > > > > > > is
> > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > initialization
> > > > > > for
> > > > > > > > > > catalog
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > > inaccurate. It
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > > registration. I
> > > > > > have
> > > > > > > > > > already
> > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > > Shammon
> > > > > FY <
> > > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > > `registerCatalog`
> > > > > > > > method
> > > > > > > > > in
> > > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > confuse users whether a method
> > > named
> > > > > > > > > `addCatalog`
> > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the
> memory
> > > > > catalog
> > > > > > > is a
> > > > > > > > > > > > `cache`,
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > > of `cache` should not be
> exposed
> > to
> > > > > > users.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I found there is a
> pre-discussion
> > > [1]
> > > > > for
> > > > > > > > this
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion
> > of
> > > that
> > > > > > > > > > discussion
> > > > > > > > > > > is
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an
> interface
> > > and
> > > > > > > > implement
> > > > > > > > > it
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > > system.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think there is a gap between
> > the
> > > > > > current
> > > > > > > > FLIP
> > > > > > > > > > > > design
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > > > > discussion
> > > > > > > in
> > > > > > > > > > thread
> > > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > >
> https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM
> > > yuxia <
> > > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this
> > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > > catalog
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll
> be
> > > > > better
> > > > > > to
> > > > > > > > > > explain
> > > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > these methods and give a
> brief
> > > > > example
> > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > > `addCatalog`
> > > > > > > still
> > > > > > > > > > looks
> > > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > > catalog to
> > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > > personal
> > > > > > > > > opinion,
> > > > > > > > > > > you
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> > statement,
> > > > > which
> > > > > > > > method
> > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > > `addCatalog`?
> > > > > I'm
> > > > > > > > > > wondering
> > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > > stement.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > > > > provide a
> > > > > > > > > default
> > > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > interface
> > `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > > registration
> > > > > for
> > > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog
> with
> > > CREATE
> > > > > > > > > CATALOG,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> by
> > > > > default.
> > > > > > > > > > > > > > > > > > > > > > > > If asynchronous registration
> is
> > > the
> > > > > > > default
> > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > > synchronous
> > > > > > > > > > registration
> > > > > > > > > > > > just
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog`
> > and
> > > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > > registration,
> > > > > it
> > > > > > > may
> > > > > > > > > well
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > > executes successfully, but
> then
> > > the
> > > > > > > > following
> > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to
> > > open.
> > > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> > which
> > > > > > should
> > > > > > > be
> > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > > registration, I
> > > > > > > > > > would
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > > will be an executor to open
> or
> > > > > register
> > > > > > > > > catalog
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > > > > initialized
> > > > > > > > if
> > > > > > > > > it
> > > > > > > > > > > has
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > > process
> > > > > > fails,
> > > > > > > > > these
> > > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > > initialization
> > > > > > > for
> > > > > > > > > > > catalog
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > > jinfeng1995@gmail.com
> > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> > dev@flink.apache.org
> > > >
> > > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > > Support
> > > > > > > > > persistence
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all
> for
> > > your
> > > > > > > > interest
> > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog
> with
> > > the
> > > > > same
> > > > > > > > name
> > > > > > > > > > that
> > > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > > point.
> > > > > > Based
> > > > > > > on
> > > > > > > > > my
> > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > > understanding, the
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and
> > > have
> > > > > the
> > > > > > > > > highest
> > > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There are three methods that
> > can
> > > > > affect
> > > > > > > the
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method puts the catalog
> > > instance
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> catalog
> > > > > > instance
> > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method first retrieves
> the
> > > > > > > > corresponding
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > > catalogs. If
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > > configuration
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > > > > initialized
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>
> catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The following two methods
> only
> > > modify
> > > > > > the
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> > catalogName,
> > > > > > > > Map<String,
> > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This method saves the
> > properties
> > > to
> > > > > the
> > > > > > > > > > > > catalogStore
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > > whether there is a
> catalogName
> > > with
> > > > > the
> > > > > > > > same
> > > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > > This method removes the
> > specified
> > > > > > > > > configuration
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> > catalogStore.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > > conflict
> > > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > > catalogName
> > > > > > > > already
> > > > > > > > > > > > exists
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > > Map<String,
> > > > > > > > > > Catalog>,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > succeed and be directly saved
> > to
> > > the
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > > catalogName
> > > > > > > > already
> > > > > > > > > > > > exists
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> > Map<String,
> > > > > > > Catalog>,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog)
> > > method
> > > > > > will
> > > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > > catalogName
> > > > > > > > already
> > > > > > > > > > > > exists
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > > addCatalog(String
> > > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > String> properties) method
> can
> > > > > directly
> > > > > > > > save
> > > > > > > > > > the
> > > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > > getCatalog(String
> > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > > initialization
> > > > > > > > > > because
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > > catalog instance already
> exists
> > > in
> > > > > > > catalogs
> > > > > > > > > and
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > > corresponding to
> > > > > > the
> > > > > > > > > > original
> > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse
> users
> > > that
> > > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)`
> > in
> > > the
> > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > > catalogName,
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This could potentially lead
> to
> > > > > > > confusion. I
> > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > > addCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > > previously
> > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > > > > instance,
> > > > > > > > > adding
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > > registerCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > > result in
> > > > > > lazy
> > > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > > encryption
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> > necessary,
> > > but
> > > > > > > > perhaps
> > > > > > > > > > the
> > > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > > > > CatalogStore
> > > > > > > > > when
> > > > > > > > > > > the
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > > FileCatalogStore
> > > > > > > > > could
> > > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > > it is saved to a file, while
> > the
> > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> > have a
> > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > > Catalog
> > > > > > > > > > > configurations
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > > implement
> > > > > > > > > asynchronous
> > > > > > > > > > > > > loading
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > > additional
> > > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > > registration
> > > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> > concern.
> > > > > When
> > > > > > > > > > executing
> > > > > > > > > > > > DDL,
> > > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > > specific
> > > > > > > > Catalog,
> > > > > > > > > > the
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > > will be initialized if it has
> > not
> > > > > been
> > > > > > > > > > > initialized
> > > > > > > > > > > > > yet.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > initialization process fails,
> > > these
> > > > > > > > > statements
> > > > > > > > > > > will
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String,
> Catalog>
> > > > > catalogs
> > > > > > > and
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have
> the
> > > same
> > > > > > > > catalog
> > > > > > > > > > > name,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > > > > referring
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> > Catalog
> > > > > that
> > > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > most
> > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > > subsequent
> > > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at
> 10:58 PM
> > > > > Xianxun
> > > > > > > Ye
> > > > > > > > <
> > > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this
> Flip,
> > > I do
> > > > > > > > believe
> > > > > > > > > > this
> > > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work
> for
> > > also
> > > > > > > > manages a
> > > > > > > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> > slow
> > > > > > because
> > > > > > > of
> > > > > > > > > > > > loading a
> > > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the
> > > user's
> > > > > > > Flink
> > > > > > > > > SQL
> > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs
> > to
> > > > > > improve
> > > > > > > > the
> > > > > > > > > > > speed
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I
> have
> > > some
> > > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog
> > with
> > > > > > CREATE
> > > > > > > > > > CATALOG,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > > registration method is used
> by
> > > > > default.
> > > > > > > > What
> > > > > > > > > if
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > > > > asynchronous
> > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > > question
> > > > > with
> > > > > > > > > > Shammon’s
> > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs
> > and
> > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> > same
> > > > > > catalog
> > > > > > > > > name,
> > > > > > > > > > > but
> > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > For
> > > > > > example:
> > > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > > hiveCatalog
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should
> define
> > > > > > `catalogs`
> > > > > > > > and
> > > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang
> Ruan
> > <
> > > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Feng.
Thanks for the update.
The current design looks good to me. I have some minor comments.
1. The `CatalogStore` need the `open`/`close` methods to init or close the
resource. For example, when storing the information in MySQL, the store
needs to open and close the connections.
2. The `getCatalog` is misspelled as `optionalDescriptor`.
3. About the usage in the sql gateway.
Considering the usage in sql gateway, the sql gateway may create a
CatalogStore for each session.
If we are using the MySqlCatalogStore, there would be so many connections.
How can we reuse the connection among these sessions?
I think sql gateway need to maintain a connection pool in
the CatalogStoreFactory and each session get its own connection from the
pool when it is created.
Then the `CatalogStoreFactory` may need the `open`/`close` methods to init
or close its resource.
Or is there a better way?
Best,
Hang
Feng Jin <ji...@gmail.com> 于2023年6月2日周五 14:45写道：
> Thanks Jingsong.
>
> > Just naming, maybe `createCatalog` in TableEnv
>
> +1 For this, I have already updated FLIP.
>
>
> Best,
> Feng
>
>
> On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com>
> wrote:
>
> > Thanks Feng,
> >
> > Just naming, maybe `createCatalog` in TableEnv, I can see many
> > functions are converted to createxxx from registerxxx.
> >
> > On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com> wrote:
> > >
> > > Hi jark, thanks for your suggestion.
> > >
> > > > 1. How to register the CatalogStore for Table API? I think the
> > > CatalogStore should be immutable once TableEnv is created. Otherwise,
> > there
> > > might be some data inconsistencies when CatalogStore is changed.
> > >
> > > Yes, We should initialize the CatalogStore when creating the TableEnv.
> > > Therefore, my current proposal is to add a method to configure the
> > > CatalogStore in EnvironmentSettings.
> > >
> > > // Initialize a catalog Store
> > > CatalogStore catalogStore = new FileCatalogStore("");
> > >
> > > // set up the Table API
> > > final EnvironmentSettings settings =
> > > EnvironmentSettings.newInstance().inBatchMode()
> > > .withCatalogStore(catalogStore)
> > > .build();
> > >
> > > final TableEnvironment tableEnv =
> > TableEnvironment.create(settings);
> > >
> > >
> > > > 2. Why does the CatalogStoreFactory interface only have a default
> > method,
> > > not an interface method?
> > >
> > > Sorry, While I did refer to the Catalog interface, I agree that as a
> new
> > > interface, the CatalogStoreFactory should not have a default method but
> > an
> > > interface method. I have already modified the interface.
> > >
> > > > 3. Please mention the alternative API in Javadoc for the
> > > deprecated`registerCatalog`.
> > > > 4. In the "Compatibility" section, would be better to mention the
> > changed
> > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > persisted catalog store) is used.
> > >
> > > Thanks for the suggestion, I have updated the FLIP [1].
> > >
> > >
> > > [1].
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > This is a useful FLIP. Thanks for starting this discussion.
> > > > The current design looks pretty good to me. I just have some minor
> > > > comments.
> > > >
> > > > 1. How to register the CatalogStore for Table API? I think the
> > CatalogStore
> > > > should be immutable once TableEnv is created. Otherwise, there might
> be
> > > > some data inconsistencies when CatalogStore is changed.
> > > >
> > > > 2. Why does the CatalogStoreFactory interface only have a default
> > method,
> > > > not an interface method?
> > > >
> > > > 3. Please mention the alternative API in Javadoc for the deprecated
> > > > `registerCatalog`.
> > > >
> > > > 4. In the "Compatibility" section, would be better to mention the
> > changed
> > > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > > persisted catalog store) is used.
> > > >
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> > > >
> > > > > Hi ， thanks all for reviewing the flip.
> > > > >
> > > > > @Ron
> > > > >
> > > > > > Regarding the CatalogStoreFactory#createCatalogStore method, do
> we
> > > > need
> > > > > to provide a default implementation?
> > > > >
> > > > > Yes, we will provide a default InMemoryCatalogStoreFactory to
> create
> > an
> > > > > InMemoryCatalogStore.
> > > > >
> > > > > > If we get a Catalog from CatalogStore, after initializing it,
> > whether
> > > > we
> > > > > put it in Map<String, Catalog> catalogs again?
> > > > >
> > > > > Yes, in the current design, catalogs are stored as snapshots, and
> > once
> > > > > initialized, the Catalog will be placed in the Map<String, Catalog>
> > > > > catalogs.
> > > > > Subsequently, the Map<String, Catalog> catalogs will be the primary
> > > > source
> > > > > for obtaining the corresponding Catalog.
> > > > >
> > > > > > how about renaming them to `catalog.store.type` and
> > > > > `catalog.store.path`?
> > > > >
> > > > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > > > strange. I
> > > > > will update the FLIP.
> > > > >
> > > > >
> > > > >
> > > > > @Shammon
> > > > >
> > > > > Thank you for the review. I have made the necessary corrections.
> > > > > Regarding the modifications made to the Public Interface, I have
> also
> > > > > included the relevant changes to the `TableEnvironment`.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > >
> > > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi feng,
> > > > > >
> > > > > > Thanks for updating, I have some minor comments
> > > > > >
> > > > > > 1. The modification of `CatalogManager` should not be in `Public
> > > > > > Interfaces`, it is not a public interface.
> > > > > >
> > > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > > `CatalogStoreFactory`
> > > > > >
> > > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > > catalogStore.get(catalogName);`
> > > > > >
> > > > > > Best,
> > > > > > Shammon FY
> > > > > >
> > > > > >
> > > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi, Feng
> > > > > > >
> > > > > > > Thanks for driving this FLIP, this proposal is very useful for
> > > > catalog
> > > > > > > management.
> > > > > > > I have some small questions:
> > > > > > >
> > > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method,
> > do we
> > > > > > need
> > > > > > > to provide a default implementation?
> > > > > > > 2. If we get Catalog from CatalogStore, after initializing it,
> > > > whether
> > > > > we
> > > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > > >
> > > > > > > Best,
> > > > > > > Ron
> > > > > > >
> > > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > > >
> > > > > > > > Hi yuxia
> > > > > > > >
> > > > > > > > > But from the code in Proposed Changes, once we register
> the
> > > > > Catalog,
> > > > > > > we
> > > > > > > > initialize it and open it. right?
> > > > > > > >
> > > > > > > > Yes, In order to avoid inconsistent semantics of the original
> > > > CREATE
> > > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > > registerCatalog
> > > > > so
> > > > > > > > that parameter validation can be performed.
> > > > > > > >
> > > > > > > > In the current design, lazy initialization is mainly
> reflected
> > in
> > > > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > > > configurations,
> > > > > > > > only initialization is required in getCatalog.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> > luoyuxia@alumni.sjtu.edu.cn
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Feng.
> > > > > > > > > I'm trying to understanding the meaning of *lazy
> > initialization*.
> > > > > If
> > > > > > > i'm
> > > > > > > > > wrong, please correct me.
> > > > > > > > >
> > > > > > > > > IIUC, lazy initialization means only you need to access the
> > > > > catalog,
> > > > > > > then
> > > > > > > > > you initialize it. But from the code in Proposed Changes,
> > once we
> > > > > > > > register
> > > > > > > > > the Catalog,
> > > > > > > > > we initialize it and open it. right?
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Yuxia
> > > > > > > > >
> > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > > > configuration
> > > > > > > > > and asynchronous registration
> > > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > > >
> > > > > > > > > One of the major changes is that current design will
> provide
> > > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> > cache,
> > > > > which
> > > > > > > > means
> > > > > > > > > once it has been initialized, any changes done by other
> > sessions
> > > > > will
> > > > > > > not
> > > > > > > > > affect it. Point 6 described follow-up options for further
> > > > > > improvement.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> > jinfeng1995@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all, I would like to update you on the latest progress
> > of
> > > > the
> > > > > > > FLIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> > ShengKai
> > > > > Fang
> > > > > > > > and I
> > > > > > > > > > had an offline discussion regarding the overall solution
> > for
> > > > > Flink
> > > > > > > > > > CatalogStore. We have reached a consensus and I have
> > updated
> > > > the
> > > > > > > final
> > > > > > > > > > solution in FLIP.
> > > > > > > > > >
> > > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > > Introduce CatalogDescriptor to store catalog
> > configuration
> > > > > > similar
> > > > > > > > to
> > > > > > > > > > TableDescriptor.
> > > > > > > > > > 2.
> > > > > > > > > >
> > > > > > > > > > The two key functions of CatalogStore - void
> > > > > storeCatalog(String
> > > > > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > > > > getCatalog(String)
> > > > > > > > > > will both use CatalogDescriptor instead of Catalog
> > instance.
> > > > > > This
> > > > > > > > way,
> > > > > > > > > > CatalogStore will only be responsible for saving and
> > > > > retrieving
> > > > > > > > > catalog
> > > > > > > > > > configurations without having to initialize catalogs.
> > > > > > > > > > 3.
> > > > > > > > > >
> > > > > > > > > > The default registerCatalog(String catalogName,
> Catalog
> > > > > catalog)
> > > > > > > > > > function in CatalogManager will be marked as
> deprecated.
> > > > > > > > > > 4.
> > > > > > > > > >
> > > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > > CatalogDescriptor
> > > > > > > > > > catalog) will be added to serve as the default
> > registration
> > > > > > > function
> > > > > > > > > for
> > > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > > 5.
> > > > > > > > > >
> > > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> > remain
> > > > > > > unchanged
> > > > > > > > > and
> > > > > > > > > > save initialized catalogs.This means that deletion
> > > > operations
> > > > > > from
> > > > > > > > one
> > > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > > 6.
> > > > > > > > > >
> > > > > > > > > > To support multi-session synchronization scenarios for
> > > > > deletions
> > > > > > > > later
> > > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > > configurable.There
> > > > > > > may
> > > > > > > > > be
> > > > > > > > > > three possible situations:
> > > > > > > > > >
> > > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > > >
> > > > > > > > > > b.Introduction of LRU cache logic which can
> > automatically
> > > > > clear
> > > > > > > > > > long-unused catalogs.
> > > > > > > > > >
> > > > > > > > > > c.No caching of any instances; each call to getCatalog
> > > > > creates a
> > > > > > > new
> > > > > > > > > > instance.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This is the document for discussion:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > > >
> > > > > > > > > > This is the final proposal document:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thank you very much for your attention and suggestions on
> > this
> > > > > > > FLIP. A
> > > > > > > > > > special thanks to Hang Ruan for his suggestions on the
> > entire
> > > > > > design
> > > > > > > > and
> > > > > > > > > > organizing offline discussions.
> > > > > > > > > >
> > > > > > > > > > If you have any further suggestions or feedback about
> this
> > FLIP
> > > > > > > please
> > > > > > > > > feel
> > > > > > > > > > free to share.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for improving the FLIP. It looks good to me. We
> > could
> > > > > > still
> > > > > > > > > > > reconsider in the future how to provide more common
> > built-in
> > > > > > cache
> > > > > > > > > > > functionality in CatalogManager so that not every
> > > > CatalogSotre
> > > > > > > > > > > implementation has to take care of it.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > >
> > > > > > > > > > > > > There might be more such issues. I would suggest
> you
> > > > > > > completely
> > > > > > > > > walk
> > > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > > >
> > > > > > > > > > > > I am very sorry for my carelessness and at the same
> > time, I
> > > > > > > greatly
> > > > > > > > > > > > appreciate your careful review. I have thoroughly
> > checked
> > > > the
> > > > > > > > entire
> > > > > > > > > > FLIP
> > > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > If I am not mistaken, with the current FLIP
> design,
> > > > > > > > > CatalogManager
> > > > > > > > > > > > could work without Optional CatalogStore being
> > configured.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, in the original design, CatalogStore was not
> > necessary
> > > > > > > because
> > > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to
> > store
> > > > > > > catalog
> > > > > > > > > > > > instances.
> > > > > > > > > > > > However, this caused inconsistency issues.
> Therefore, I
> > > > > > modified
> > > > > > > > this
> > > > > > > > > > > part
> > > > > > > > > > > > of the design and removed Map<String, Catalog>
> catalogs
> > > > from
> > > > > > > > > > > > CatalogManager.
> > > > > > > > > > > > At the same time, InMemoryCatalog will serve as the
> > > > default
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > to save catalogs in memory and replace the
> > functionality of
> > > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > > The previous plan that kept
> > Map<String,Catalog>catalogs has
> > > > > > been
> > > > > > > > > moved
> > > > > > > > > > to
> > > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > There are still many places contain inconsistent
> > content,
> > > > > > e.g.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > > 2. The java comment of the method
> > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog catalog) in CatalogManager does not tell
> > what the
> > > > > > > method
> > > > > > > > is
> > > > > > > > > > > > doing.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > There might be more such issues. I would suggest
> you
> > > > > > completely
> > > > > > > > > walk
> > > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > > >
> > > > > > > > > > > > > About the inMemoryCatalogStore, do you mean that
> you
> > will
> > > > > > build
> > > > > > > > the
> > > > > > > > > > > cache
> > > > > > > > > > > > > functionality in the CatalogStore? This is a very
> > > > different
> > > > > > > > design
> > > > > > > > > > > > concept
> > > > > > > > > > > > > from what the current FLIP described. If I am not
> > > > mistaken,
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > current FLIP design, CatalogManager could work
> > without
> > > > > > Optional
> > > > > > > > > > > > > CatalogStore being configured. That is the reason
> > why I
> > > > > > > mentioned
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > last email that the example code wrt the Optional
> > > > > > CatalogStore
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > > correct.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you very much for the careful review. I
> have
> > > > > already
> > > > > > > made
> > > > > > > > > the
> > > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > > multi-instance
> > > > > > > > > > scenarios, I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Currently, there are conflicts in multiple
> > scenarios
> > > > with
> > > > > > the
> > > > > > > > > > current
> > > > > > > > > > > > > > design. I am thinking whether we should remove
> > > > > 'Map<String,
> > > > > > > > > > Catalog>'
> > > > > > > > > > > > and
> > > > > > > > > > > > > > make Cache the default behavior of
> > > > InMemoryCatalogStore.
> > > > > > This
> > > > > > > > > way,
> > > > > > > > > > > > users
> > > > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > > > multi-instance
> > > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for working on the FLIP. There are still
> > some
> > > > > NIT
> > > > > > > > issues
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > like:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been
> > used
> > > > as
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > instead of Optional in the code example. It
> > should be
> > > > > > fine
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > it
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > pseudo code for now and update it after you
> > submit
> > > > the
> > > > > > PR.
> > > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in
> the
> > > > > > rejected
> > > > > > > > > > section
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > should be persistContext(...) to keep it
> > consistent.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Speaking of the conflict issues in the
> > multi-instance
> > > > > > > > > scenarios,
> > > > > > > > > > I
> > > > > > > > > > > am
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > > Map<String,
> > > > > > > > Catalog>
> > > > > > > > > > > > > catalogs
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > used as a cache, it should be invalid, once the
> > > > related
> > > > > > > > catalog
> > > > > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > removed from the CatalogStore by another
> > instance.
> > > > Did
> > > > > I
> > > > > > > miss
> > > > > > > > > > > > > something?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > > I think this is a good function name, I have
> > > > updated
> > > > > it
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Some common cache features should be
> > > > implemented
> > > > > > > > > > > > > > > > Thank you for the suggestion. If alternative
> 1
> > > > turns
> > > > > > out
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > appropriate later, I will improve this part
> of
> > the
> > > > > > > content.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As the above discussion moves forward, the
> > > > option 2
> > > > > > > > > solution
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > > Yes, after discussing with Shammon offline,
> we
> > > > think
> > > > > > that
> > > > > > > > > > > solution
> > > > > > > > > > > > 2
> > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > be more suitable and also avoid any
> > inconsistency
> > > > > > issues.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There are some inconsistent descriptions in
> > the
> > > > > > > content.
> > > > > > > > > > Would
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > > I will do my best to improve the document and
> > > > > > appreciate
> > > > > > > > your
> > > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > can you put the unselected option in
> > `Rejected
> > > > > > > > > Alternatives`
> > > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > > Alternatives`.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I found there are two options in `Proposed
> > > > > Changes`,
> > > > > > > can
> > > > > > > > > you
> > > > > > > > > > > put
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > unselected option in `Rejected
> > Alternatives`? I
> > > > > think
> > > > > > > > this
> > > > > > > > > > may
> > > > > > > > > > > > help
> > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> > > > confused
> > > > > > > after
> > > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog()
> and
> > > > > > > > addCatalog()
> > > > > > > > > > have
> > > > > > > > > > > > no
> > > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > > difference based on their names. One of
> > them is
> > > > > > > > > responsible
> > > > > > > > > > > for
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > is
> > > > > > > > > > > used
> > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > and catalogStore is used for data
> > persistence.
> > > > I
> > > > > > > would
> > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > > their purpose conceptually and clearly in
> > the
> > > > > FLIP.
> > > > > > > > Some
> > > > > > > > > > > common
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > features should be implemented, i.e. data
> > in
> > > > the
> > > > > > > cache
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> > should be
> > > > > > found
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > store
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the cache(either it has been initialized
> > or it
> > > > > will
> > > > > > > be
> > > > > > > > > lazy
> > > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > > for the same catalog name. The
> consistency
> > will
> > > > > be
> > > > > > > > taken
> > > > > > > > > > care
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > > 3. As the above discussion moves forward,
> > the
> > > > > > option
> > > > > > > 2
> > > > > > > > > > > solution
> > > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > like a replacement of option 1, because,
> > afaiu,
> > > > > > > issues
> > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > previously with option 1 are not solved
> > yet. Do
> > > > > you
> > > > > > > > still
> > > > > > > > > > > want
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > > both options and ask for suggestions for
> > both
> > > > of
> > > > > > > them?
> > > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there are
> > some
> > > > > > > > > inconsistent
> > > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the content. Would you like to clean
> them
> > up?
> > > > > > > Thanks!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you for your response, and I
> > completely
> > > > > > agree
> > > > > > > > > with
> > > > > > > > > > > your
> > > > > > > > > > > > > > point
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > > Initially, I may have over complicated
> > the
> > > > > whole
> > > > > > > > issue.
> > > > > > > > > > > First
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > > we need to consider the persistence of
> > the
> > > > > > > Catalog's
> > > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > > If we only need to provide persistence
> > for
> > > > > > Catalog
> > > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> > Catalog
> > > > > > > > interface.
> > > > > > > > > > > > > > > > > > > This method can convert a Catalog
> > instance
> > > > to a
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties, and the default
> > implementation
> > > > will
> > > > > > > throw
> > > > > > > > > an
> > > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > > * Returns a map containing the
> > properties
> > > > of
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > > * @return a map containing the
> > properties
> > > > of
> > > > > > the
> > > > > > > > > > catalog
> > > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > > * @throws
> > UnsupportedOperationException if
> > > > > the
> > > > > > > > > > > > implementing
> > > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > > * the default implementation of this
> > > > method
> > > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > > default Map<String, String>
> > toProperties()
> > > > {
> > > > > > > > > > > > > > > > > > > throw new
> > > > > > UnsupportedOperationException("Please
> > > > > > > > > > > implement
> > > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > > CatalogStore,
> > > > > > the
> > > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will be called when registering a
> Catalog
> > > > > > instance
> > > > > > > > with
> > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > Catalog
> > > > > > > catalog).
> > > > > > > > > The
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> > String>
> > > > > > > properties
> > > > > > > > > and
> > > > > > > > > > > > saved
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances
> > do
> > > > not
> > > > > > > > > implement
> > > > > > > > > > > this
> > > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. If the user does not use a
> > CatalogStore,
> > > > > the
> > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > will not be called, ensuring
> consistency
> > with
> > > > > the
> > > > > > > > > > original
> > > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > > > > catalogs
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > > we can start from the Catalog itself
> and
> > > > > provide
> > > > > > a
> > > > > > > > > > > dedicated
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > implementation for lazy loading, such
> as
> > > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> > implements
> > > > > > > Catalog
> > > > > > > > {
> > > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String,
> String>
> > > > > > > properties)
> > > > > > > > {
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon
> > FY <
> > > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I have no questions about the
> > functionality
> > > > > of
> > > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > > `registerCatalog`
> > > > > > > > or
> > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > `CatalogManager`. The implementation
> in
> > > > Trino
> > > > > > is
> > > > > > > > also
> > > > > > > > > > the
> > > > > > > > > > > > > same,
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has
> one
> > > > > > > > > `createCatalog`,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > > > > `registerCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > save a catalog in `Map<String,
> Catalog>
> > > > > > catalogs`
> > > > > > > > or
> > > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> > before,
> > > > the
> > > > > > > > > > `Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> > `CatalogManager`.
> > > > How
> > > > > > > about
> > > > > > > > > > > saving
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > > > `CatalogStore
> > > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng
> > Jin <
> > > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am very
> > sorry
> > > > > for
> > > > > > > the
> > > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > > > discussion.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> > [1]
> > > > for
> > > > > > > this
> > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> > discussion
> > > > > > before.
> > > > > > > > > > > However,
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > whole solution, I found that the
> > logic of
> > > > > > > > > > > CatalogManager
> > > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > > need to change much. *We cannot
> only
> > > > > persist
> > > > > > > > > Catalog
> > > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > > so exposing only
> > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> > because
> > > > in
> > > > > > the
> > > > > > > > end
> > > > > > > > > we
> > > > > > > > > > > > still
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > > the configurations corresponding to
> > the
> > > > > > Catalog
> > > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > > decided to introduce the
> CatalogStore
> > > > > > interface
> > > > > > > > for
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > persistence. Regarding this part, I
> > also
> > > > > took
> > > > > > > > > > > inspiration
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > catalog
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > name
> > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I
> will
> > > > > provide
> > > > > > a
> > > > > > > > > > detailed
> > > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > code examples for this part, and
> add
> > it
> > > > to
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > `addCatalog`
> > > > > still
> > > > > > > > looks
> > > > > > > > > > > > confused
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > this method is for storing catalog
> to
> > > > > > > > CatalogStore,
> > > > > > > > > > how
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> personal
> > > > > > opinion,
> > > > > > > > you
> > > > > > > > > > can
> > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive
> to
> > me.
> > > > I
> > > > > > > don't
> > > > > > > > > see
> > > > > > > > > > > any
> > > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement,
> > which
> > > > > > method
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> > I'm
> > > > > > > wondering
> > > > > > > > > > > whether
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> > design
> > > > was
> > > > > to
> > > > > > > add
> > > > > > > > > it
> > > > > > > > > > > > > directly
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore, but this would
> disrupt
> > the
> > > > > > > existing
> > > > > > > > > > > logic.
> > > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > we can do both: save the
> > configuration to
> > > > > the
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > > the Catalog instance at the same
> time
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> > provide a
> > > > > > > default
> > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > I think it is necessary, otherwise
> we
> > > > would
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > > > introduce
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > > Map to store the configuration for
> > lazy
> > > > > > > loading.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> registration
> > for
> > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > I don't think
> registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > made into an asynchronous interface
> > > > because
> > > > > > > > Catalog
> > > > > > > > > > is
> > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > initialization
> > > > > for
> > > > > > > > > catalog
> > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > Indeed, my description was
> > inaccurate. It
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > lazy
> > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > instead of asynchronous
> > registration. I
> > > > > have
> > > > > > > > > already
> > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> > Shammon
> > > > FY <
> > > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think if there is a
> > `registerCatalog`
> > > > > > > method
> > > > > > > > in
> > > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > confuse users whether a method
> > named
> > > > > > > > `addCatalog`
> > > > > > > > > > or
> > > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> > > > catalog
> > > > > > is a
> > > > > > > > > > > `cache`,
> > > > > > > > > > > > I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > > of `cache` should not be exposed
> to
> > > > > users.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> > [1]
> > > > for
> > > > > > > this
> > > > > > > > > > FLIP.
> > > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion
> of
> > that
> > > > > > > > > discussion
> > > > > > > > > > is
> > > > > > > > > > > > to
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface
> > and
> > > > > > > implement
> > > > > > > > it
> > > > > > > > > > for
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > as memory, file and external
> > system.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think there is a gap between
> the
> > > > > current
> > > > > > > FLIP
> > > > > > > > > > > design
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > > > discussion
> > > > > > in
> > > > > > > > > thread
> > > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM
> > yuxia <
> > > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this
> > FLIP.
> > > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> > catalog
> > > > > with
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > name
> > > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> > > > better
> > > > > to
> > > > > > > > > explain
> > > > > > > > > > > it
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > these methods and give a brief
> > > > example
> > > > > in
> > > > > > > > this
> > > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> > `addCatalog`
> > > > > > still
> > > > > > > > > looks
> > > > > > > > > > > > > confused
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > > this method is for storing
> > catalog to
> > > > > > > > > > CatalogStore,
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> > personal
> > > > > > > > opinion,
> > > > > > > > > > you
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG
> statement,
> > > > which
> > > > > > > method
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> > `addCatalog`?
> > > > I'm
> > > > > > > > > wondering
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> > stement.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > > > provide a
> > > > > > > > default
> > > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > interface
> `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> > registration
> > > > for
> > > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > When creating a catalog with
> > CREATE
> > > > > > > > CATALOG,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > > default.
> > > > > > > > > > > > > > > > > > > > > > > If asynchronous registration is
> > the
> > > > > > default
> > > > > > > > > > > behavior,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > > that user can switch to
> > synchronous
> > > > > > > > > registration
> > > > > > > > > > > just
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog`
> and
> > > > > > > > > > `registerCatalog`
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> > registration,
> > > > it
> > > > > > may
> > > > > > > > well
> > > > > > > > > > > that
> > > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > > executes successfully, but then
> > the
> > > > > > > following
> > > > > > > > > > > CREATE
> > > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to
> > open.
> > > > > > > > > > > > > > > > > > > > > > > I think it's a break change
> which
> > > > > should
> > > > > > be
> > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > > registration, I
> > > > > > > > > would
> > > > > > > > > > > > like
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > > will be an executor to open or
> > > > register
> > > > > > > > catalog
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > > > initialized
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > has
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> > process
> > > > > fails,
> > > > > > > > these
> > > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > > initialization
> > > > > > for
> > > > > > > > > > catalog
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > > jinfeng1995@gmail.com
> > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <
> dev@flink.apache.org
> > >
> > > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> > 3:27:45
> > > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> > Support
> > > > > > > > persistence
> > > > > > > > > > of
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for
> > your
> > > > > > > interest
> > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog with
> > the
> > > > same
> > > > > > > name
> > > > > > > > > that
> > > > > > > > > > > > > exists
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> > point.
> > > > > Based
> > > > > > on
> > > > > > > > my
> > > > > > > > > > > > current
> > > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > > will
> > > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and
> > have
> > > > the
> > > > > > > > highest
> > > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There are three methods that
> can
> > > > affect
> > > > > > the
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method puts the catalog
> > instance
> > > > > > into
> > > > > > > > the
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > > > > instance
> > > > > > > > > > > > corresponding
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String
> catalogName)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > > > > corresponding
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> > catalogs. If
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > > configuration
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > > > initialized
> > > > > > > > > Catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The following two methods only
> > modify
> > > > > the
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String
> catalogName,
> > > > > > > Map<String,
> > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This method saves the
> properties
> > to
> > > > the
> > > > > > > > > > > catalogStore
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > > whether there is a catalogName
> > with
> > > > the
> > > > > > > same
> > > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> > catalogName)
> > > > > > > > > > > > > > > > > > > > > > > This method removes the
> specified
> > > > > > > > configuration
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > > catalogName in the
> catalogStore.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The following are possible
> > conflict
> > > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> > catalogName
> > > > > > > already
> > > > > > > > > > > exists
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > > Map<String,
> > > > > > > > > Catalog>,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > succeed and be directly saved
> to
> > the
> > > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> > catalogName
> > > > > > > already
> > > > > > > > > > > exists
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogStore and the
> Map<String,
> > > > > > Catalog>,
> > > > > > > > the
> > > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog)
> > method
> > > > > will
> > > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> > catalogName
> > > > > > > already
> > > > > > > > > > > exists
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > > addCatalog(String
> > > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > String> properties) method can
> > > > directly
> > > > > > > save
> > > > > > > > > the
> > > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > > getCatalog(String
> > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > > initialization
> > > > > > > > > because
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > > catalog instance already exists
> > in
> > > > > > catalogs
> > > > > > > > and
> > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > > unregisterCatalog(String
> > > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > remove the instance
> > corresponding to
> > > > > the
> > > > > > > > > original
> > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think it will confuse users
> > that
> > > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)`
> in
> > the
> > > > > > > > > `Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> > catalogName,
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > > > > confusion. I
> > > > > > > > > > suggest
> > > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > > addCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > > String> properties), as
> > previously
> > > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > > > instance,
> > > > > > > > adding
> > > > > > > > > > the
> > > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > > registerCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> > result in
> > > > > lazy
> > > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > > encryption
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think encryption is
> necessary,
> > but
> > > > > > > perhaps
> > > > > > > > > the
> > > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > > > CatalogStore
> > > > > > > > when
> > > > > > > > > > the
> > > > > > > > > > > > data
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > > FileCatalogStore
> > > > > > > > could
> > > > > > > > > > > > encrypt
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > > it is saved to a file, while
> the
> > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it is necessary to
> have a
> > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > as
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> > Catalog
> > > > > > > > > > configurations
> > > > > > > > > > > in
> > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> > implement
> > > > > > > > asynchronous
> > > > > > > > > > > > loading
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > would need to introduce
> > additional
> > > > > cache.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> > registration
> > > > > > failed?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This is also a critical
> concern.
> > > > When
> > > > > > > > > executing
> > > > > > > > > > > DDL,
> > > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > > statements that reference a
> > specific
> > > > > > > Catalog,
> > > > > > > > > the
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > > will be initialized if it has
> not
> > > > been
> > > > > > > > > > initialized
> > > > > > > > > > > > yet.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > initialization process fails,
> > these
> > > > > > > > statements
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> > > > catalogs
> > > > > > and
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have the
> > same
> > > > > > > catalog
> > > > > > > > > > name,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > > > referring
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > key principle is that the
> Catalog
> > > > that
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > most
> > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > > will be given priority for
> > subsequent
> > > > > > use.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> > > > Xianxun
> > > > > > Ye
> > > > > > > <
> > > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip,
> > I do
> > > > > > > believe
> > > > > > > > > this
> > > > > > > > > > > > Flip
> > > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > > for users. The firm I work for
> > also
> > > > > > > manages a
> > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes
> slow
> > > > > because
> > > > > > of
> > > > > > > > > > > loading a
> > > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the
> > user's
> > > > > > Flink
> > > > > > > > SQL
> > > > > > > > > > > > through
> > > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs
> to
> > > > > improve
> > > > > > > the
> > > > > > > > > > speed
> > > > > > > > > > > of
> > > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I have
> > some
> > > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog
> with
> > > > > CREATE
> > > > > > > > > CATALOG,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > > default.
> > > > > > > What
> > > > > > > > if
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > > > asynchronous
> > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> > question
> > > > with
> > > > > > > > > Shammon’s
> > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs
> and
> > > > > > > > CatalogStore
> > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the
> same
> > > > > catalog
> > > > > > > > name,
> > > > > > > > > > but
> > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> For
> > > > > example:
> > > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> > hiveCatalog
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > > > > `catalogs`
> > > > > > > and
> > > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan
> <
> > > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Thanks Jingsong.
> Just naming, maybe `createCatalog` in TableEnv
+1 For this, I have already updated FLIP.
Best,
Feng
On Fri, Jun 2, 2023 at 11:32 AM Jingsong Li <ji...@gmail.com> wrote:
> Thanks Feng,
>
> Just naming, maybe `createCatalog` in TableEnv, I can see many
> functions are converted to createxxx from registerxxx.
>
> On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com> wrote:
> >
> > Hi jark, thanks for your suggestion.
> >
> > > 1. How to register the CatalogStore for Table API? I think the
> > CatalogStore should be immutable once TableEnv is created. Otherwise,
> there
> > might be some data inconsistencies when CatalogStore is changed.
> >
> > Yes, We should initialize the CatalogStore when creating the TableEnv.
> > Therefore, my current proposal is to add a method to configure the
> > CatalogStore in EnvironmentSettings.
> >
> > // Initialize a catalog Store
> > CatalogStore catalogStore = new FileCatalogStore("");
> >
> > // set up the Table API
> > final EnvironmentSettings settings =
> > EnvironmentSettings.newInstance().inBatchMode()
> > .withCatalogStore(catalogStore)
> > .build();
> >
> > final TableEnvironment tableEnv =
> TableEnvironment.create(settings);
> >
> >
> > > 2. Why does the CatalogStoreFactory interface only have a default
> method,
> > not an interface method?
> >
> > Sorry, While I did refer to the Catalog interface, I agree that as a new
> > interface, the CatalogStoreFactory should not have a default method but
> an
> > interface method. I have already modified the interface.
> >
> > > 3. Please mention the alternative API in Javadoc for the
> > deprecated`registerCatalog`.
> > > 4. In the "Compatibility" section, would be better to mention the
> changed
> > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > persisted catalog store) is used.
> >
> > Thanks for the suggestion, I have updated the FLIP [1].
> >
> >
> > [1].
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > Hi Feng,
> > >
> > > This is a useful FLIP. Thanks for starting this discussion.
> > > The current design looks pretty good to me. I just have some minor
> > > comments.
> > >
> > > 1. How to register the CatalogStore for Table API? I think the
> CatalogStore
> > > should be immutable once TableEnv is created. Otherwise, there might be
> > > some data inconsistencies when CatalogStore is changed.
> > >
> > > 2. Why does the CatalogStoreFactory interface only have a default
> method,
> > > not an interface method?
> > >
> > > 3. Please mention the alternative API in Javadoc for the deprecated
> > > `registerCatalog`.
> > >
> > > 4. In the "Compatibility" section, would be better to mention the
> changed
> > > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > > persisted catalog store) is used.
> > >
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> > >
> > > > Hi ， thanks all for reviewing the flip.
> > > >
> > > > @Ron
> > > >
> > > > > Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > > need
> > > > to provide a default implementation?
> > > >
> > > > Yes, we will provide a default InMemoryCatalogStoreFactory to create
> an
> > > > InMemoryCatalogStore.
> > > >
> > > > > If we get a Catalog from CatalogStore, after initializing it,
> whether
> > > we
> > > > put it in Map<String, Catalog> catalogs again?
> > > >
> > > > Yes, in the current design, catalogs are stored as snapshots, and
> once
> > > > initialized, the Catalog will be placed in the Map<String, Catalog>
> > > > catalogs.
> > > > Subsequently, the Map<String, Catalog> catalogs will be the primary
> > > source
> > > > for obtaining the corresponding Catalog.
> > > >
> > > > > how about renaming them to `catalog.store.type` and
> > > > `catalog.store.path`?
> > > >
> > > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > > strange. I
> > > > will update the FLIP.
> > > >
> > > >
> > > >
> > > > @Shammon
> > > >
> > > > Thank you for the review. I have made the necessary corrections.
> > > > Regarding the modifications made to the Public Interface, I have also
> > > > included the relevant changes to the `TableEnvironment`.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > >
> > > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com>
> wrote:
> > > >
> > > > > Hi feng,
> > > > >
> > > > > Thanks for updating, I have some minor comments
> > > > >
> > > > > 1. The modification of `CatalogManager` should not be in `Public
> > > > > Interfaces`, it is not a public interface.
> > > > >
> > > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > > `CatalogStoreFactory`
> > > > >
> > > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > > catalogStore.get(catalogName);`
> > > > >
> > > > > Best,
> > > > > Shammon FY
> > > > >
> > > > >
> > > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi, Feng
> > > > > >
> > > > > > Thanks for driving this FLIP, this proposal is very useful for
> > > catalog
> > > > > > management.
> > > > > > I have some small questions:
> > > > > >
> > > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method,
> do we
> > > > > need
> > > > > > to provide a default implementation?
> > > > > > 2. If we get Catalog from CatalogStore, after initializing it,
> > > whether
> > > > we
> > > > > > put it to Map<String, Catalog> catalogs again?
> > > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > > `catalog.store.type` and `catalog.store.path`?
> > > > > >
> > > > > > Best,
> > > > > > Ron
> > > > > >
> > > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > > >
> > > > > > > Hi yuxia
> > > > > > >
> > > > > > > > But from the code in Proposed Changes, once we register the
> > > > Catalog,
> > > > > > we
> > > > > > > initialize it and open it. right?
> > > > > > >
> > > > > > > Yes, In order to avoid inconsistent semantics of the original
> > > CREATE
> > > > > > > CATALOG DDL, Catalog will be directly initialized in
> > > registerCatalog
> > > > so
> > > > > > > that parameter validation can be performed.
> > > > > > >
> > > > > > > In the current design, lazy initialization is mainly reflected
> in
> > > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > > configurations,
> > > > > > > only initialization is required in getCatalog.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <
> luoyuxia@alumni.sjtu.edu.cn
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Feng.
> > > > > > > > I'm trying to understanding the meaning of *lazy
> initialization*.
> > > > If
> > > > > > i'm
> > > > > > > > wrong, please correct me.
> > > > > > > >
> > > > > > > > IIUC, lazy initialization means only you need to access the
> > > > catalog,
> > > > > > then
> > > > > > > > you initialize it. But from the code in Proposed Changes,
> once we
> > > > > > > register
> > > > > > > > the Catalog,
> > > > > > > > we initialize it and open it. right?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Yuxia
> > > > > > > >
> > > > > > > > ----- 原始邮件 -----
> > > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > > configuration
> > > > > > > > and asynchronous registration
> > > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > > >
> > > > > > > > One of the major changes is that current design will provide
> > > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a
> cache,
> > > > which
> > > > > > > means
> > > > > > > > once it has been initialized, any changes done by other
> sessions
> > > > will
> > > > > > not
> > > > > > > > affect it. Point 6 described follow-up options for further
> > > > > improvement.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <
> jinfeng1995@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all, I would like to update you on the latest progress
> of
> > > the
> > > > > > FLIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY,
> ShengKai
> > > > Fang
> > > > > > > and I
> > > > > > > > > had an offline discussion regarding the overall solution
> for
> > > > Flink
> > > > > > > > > CatalogStore. We have reached a consensus and I have
> updated
> > > the
> > > > > > final
> > > > > > > > > solution in FLIP.
> > > > > > > > >
> > > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > >
> > > > > > > > > Introduce CatalogDescriptor to store catalog
> configuration
> > > > > similar
> > > > > > > to
> > > > > > > > > TableDescriptor.
> > > > > > > > > 2.
> > > > > > > > >
> > > > > > > > > The two key functions of CatalogStore - void
> > > > storeCatalog(String
> > > > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > > > getCatalog(String)
> > > > > > > > > will both use CatalogDescriptor instead of Catalog
> instance.
> > > > > This
> > > > > > > way,
> > > > > > > > > CatalogStore will only be responsible for saving and
> > > > retrieving
> > > > > > > > catalog
> > > > > > > > > configurations without having to initialize catalogs.
> > > > > > > > > 3.
> > > > > > > > >
> > > > > > > > > The default registerCatalog(String catalogName, Catalog
> > > > catalog)
> > > > > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > > > > 4.
> > > > > > > > >
> > > > > > > > > A new function registerCatalog(String catalogName,
> > > > > > CatalogDescriptor
> > > > > > > > > catalog) will be added to serve as the default
> registration
> > > > > > function
> > > > > > > > for
> > > > > > > > > catalogs in CatalogManager.
> > > > > > > > > 5.
> > > > > > > > >
> > > > > > > > > Map<String,Catalog> catalogs in CataloManager will
> remain
> > > > > > unchanged
> > > > > > > > and
> > > > > > > > > save initialized catalogs.This means that deletion
> > > operations
> > > > > from
> > > > > > > one
> > > > > > > > > session won't synchronize with other sessions.
> > > > > > > > > 6.
> > > > > > > > >
> > > > > > > > > To support multi-session synchronization scenarios for
> > > > deletions
> > > > > > > later
> > > > > > > > > on we should make Map<String,Catalog>catalogs
> > > > configurable.There
> > > > > > may
> > > > > > > > be
> > > > > > > > > three possible situations:
> > > > > > > > >
> > > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > > >
> > > > > > > > > b.Introduction of LRU cache logic which can
> automatically
> > > > clear
> > > > > > > > > long-unused catalogs.
> > > > > > > > >
> > > > > > > > > c.No caching of any instances; each call to getCatalog
> > > > creates a
> > > > > > new
> > > > > > > > > instance.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is the document for discussion:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > > >
> > > > > > > > > This is the final proposal document:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thank you very much for your attention and suggestions on
> this
> > > > > > FLIP. A
> > > > > > > > > special thanks to Hang Ruan for his suggestions on the
> entire
> > > > > design
> > > > > > > and
> > > > > > > > > organizing offline discussions.
> > > > > > > > >
> > > > > > > > > If you have any further suggestions or feedback about this
> FLIP
> > > > > > please
> > > > > > > > feel
> > > > > > > > > free to share.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > Thanks for improving the FLIP. It looks good to me. We
> could
> > > > > still
> > > > > > > > > > reconsider in the future how to provide more common
> built-in
> > > > > cache
> > > > > > > > > > functionality in CatalogManager so that not every
> > > CatalogSotre
> > > > > > > > > > implementation has to take care of it.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > > jinfeng1995@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jing,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > >
> > > > > > > > > > > > There might be more such issues. I would suggest you
> > > > > > completely
> > > > > > > > walk
> > > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > > >
> > > > > > > > > > > I am very sorry for my carelessness and at the same
> time, I
> > > > > > greatly
> > > > > > > > > > > appreciate your careful review. I have thoroughly
> checked
> > > the
> > > > > > > entire
> > > > > > > > > FLIP
> > > > > > > > > > > and made corrections to these issues.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > > > > CatalogManager
> > > > > > > > > > > could work without Optional CatalogStore being
> configured.
> > > > > > > > > > >
> > > > > > > > > > > Yes, in the original design, CatalogStore was not
> necessary
> > > > > > because
> > > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to
> store
> > > > > > catalog
> > > > > > > > > > > instances.
> > > > > > > > > > > However, this caused inconsistency issues. Therefore, I
> > > > > modified
> > > > > > > this
> > > > > > > > > > part
> > > > > > > > > > > of the design and removed Map<String, Catalog> catalogs
> > > from
> > > > > > > > > > > CatalogManager.
> > > > > > > > > > > At the same time, InMemoryCatalog will serve as the
> > > default
> > > > > > > > > CatalogStore
> > > > > > > > > > > to save catalogs in memory and replace the
> functionality of
> > > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > > The previous plan that kept
> Map<String,Catalog>catalogs has
> > > > > been
> > > > > > > > moved
> > > > > > > > > to
> > > > > > > > > > > Rejected Alternatives.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > There are still many places contain inconsistent
> content,
> > > > > e.g.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > > 2. The java comment of the method
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog catalog) in CatalogManager does not tell
> what the
> > > > > > method
> > > > > > > is
> > > > > > > > > > > doing.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > There might be more such issues. I would suggest you
> > > > > completely
> > > > > > > > walk
> > > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > > >
> > > > > > > > > > > > About the inMemoryCatalogStore, do you mean that you
> will
> > > > > build
> > > > > > > the
> > > > > > > > > > cache
> > > > > > > > > > > > functionality in the CatalogStore? This is a very
> > > different
> > > > > > > design
> > > > > > > > > > > concept
> > > > > > > > > > > > from what the current FLIP described. If I am not
> > > mistaken,
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > current FLIP design, CatalogManager could work
> without
> > > > > Optional
> > > > > > > > > > > > CatalogStore being configured. That is the reason
> why I
> > > > > > mentioned
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > last email that the example code wrt the Optional
> > > > > CatalogStore
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > > correct.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you very much for the careful review. I have
> > > > already
> > > > > > made
> > > > > > > > the
> > > > > > > > > > > > > relevant changes.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Speaking of the conflict issues in the
> > > multi-instance
> > > > > > > > > scenarios, I
> > > > > > > > > > > am
> > > > > > > > > > > > > not
> > > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > > >
> > > > > > > > > > > > > Currently, there are conflicts in multiple
> scenarios
> > > with
> > > > > the
> > > > > > > > > current
> > > > > > > > > > > > > design. I am thinking whether we should remove
> > > > 'Map<String,
> > > > > > > > > Catalog>'
> > > > > > > > > > > and
> > > > > > > > > > > > > make Cache the default behavior of
> > > InMemoryCatalogStore.
> > > > > This
> > > > > > > > way,
> > > > > > > > > > > users
> > > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > > multi-instance
> > > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for working on the FLIP. There are still
> some
> > > > NIT
> > > > > > > issues
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > like:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been
> used
> > > as
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > instead of Optional in the code example. It
> should be
> > > > > fine
> > > > > > to
> > > > > > > > use
> > > > > > > > > > it
> > > > > > > > > > > as
> > > > > > > > > > > > > > pseudo code for now and update it after you
> submit
> > > the
> > > > > PR.
> > > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > > > > rejected
> > > > > > > > > section
> > > > > > > > > > > > which
> > > > > > > > > > > > > > should be persistContext(...) to keep it
> consistent.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Speaking of the conflict issues in the
> multi-instance
> > > > > > > > scenarios,
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > > > sure if this is the intended behaviour. If
> > > Map<String,
> > > > > > > Catalog>
> > > > > > > > > > > > catalogs
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > used as a cache, it should be invalid, once the
> > > related
> > > > > > > catalog
> > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > > removed from the CatalogStore by another
> instance.
> > > Did
> > > > I
> > > > > > miss
> > > > > > > > > > > > something?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > > I think this is a good function name, I have
> > > updated
> > > > it
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Some common cache features should be
> > > implemented
> > > > > > > > > > > > > > > Thank you for the suggestion. If alternative 1
> > > turns
> > > > > out
> > > > > > to
> > > > > > > > be
> > > > > > > > > > more
> > > > > > > > > > > > > > > appropriate later, I will improve this part of
> the
> > > > > > content.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As the above discussion moves forward, the
> > > option 2
> > > > > > > > solution
> > > > > > > > > > > looks
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > > Yes, after discussing with Shammon offline, we
> > > think
> > > > > that
> > > > > > > > > > solution
> > > > > > > > > > > 2
> > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > be more suitable and also avoid any
> inconsistency
> > > > > issues.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There are some inconsistent descriptions in
> the
> > > > > > content.
> > > > > > > > > Would
> > > > > > > > > > > you
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > > I will do my best to improve the document and
> > > > > appreciate
> > > > > > > your
> > > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > can you put the unselected option in
> `Rejected
> > > > > > > > Alternatives`
> > > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > > Alternatives`.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I found there are two options in `Proposed
> > > > Changes`,
> > > > > > can
> > > > > > > > you
> > > > > > > > > > put
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > unselected option in `Rejected
> Alternatives`? I
> > > > think
> > > > > > > this
> > > > > > > > > may
> > > > > > > > > > > help
> > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> > > confused
> > > > > > after
> > > > > > > > > > > > completely
> > > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > > > > addCatalog()
> > > > > > > > > have
> > > > > > > > > > > no
> > > > > > > > > > > > > big
> > > > > > > > > > > > > > > > > difference based on their names. One of
> them is
> > > > > > > > responsible
> > > > > > > > > > for
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > > > 2. As you mentioned that Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > is
> > > > > > > > > > used
> > > > > > > > > > > > as a
> > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > and catalogStore is used for data
> persistence.
> > > I
> > > > > > would
> > > > > > > > > > suggest
> > > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > > their purpose conceptually and clearly in
> the
> > > > FLIP.
> > > > > > > Some
> > > > > > > > > > common
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > features should be implemented, i.e. data
> in
> > > the
> > > > > > cache
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > store
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be consistent. Same Catalog instance
> should be
> > > > > found
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > store
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the cache(either it has been initialized
> or it
> > > > will
> > > > > > be
> > > > > > > > lazy
> > > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > > for the same catalog name. The consistency
> will
> > > > be
> > > > > > > taken
> > > > > > > > > care
> > > > > > > > > > > of
> > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > > 3. As the above discussion moves forward,
> the
> > > > > option
> > > > > > 2
> > > > > > > > > > solution
> > > > > > > > > > > > > looks
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > like a replacement of option 1, because,
> afaiu,
> > > > > > issues
> > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > previously with option 1 are not solved
> yet. Do
> > > > you
> > > > > > > still
> > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > > both options and ask for suggestions for
> both
> > > of
> > > > > > them?
> > > > > > > > > > > > > > > > > 4. After you updated the FLIP, there are
> some
> > > > > > > > inconsistent
> > > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the content. Would you like to clean them
> up?
> > > > > > Thanks!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you for your response, and I
> completely
> > > > > agree
> > > > > > > > with
> > > > > > > > > > your
> > > > > > > > > > > > > point
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > > Initially, I may have over complicated
> the
> > > > whole
> > > > > > > issue.
> > > > > > > > > > First
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > > we need to consider the persistence of
> the
> > > > > > Catalog's
> > > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > > If we only need to provide persistence
> for
> > > > > Catalog
> > > > > > > > > > > > Configuration,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > add a toConfiguration method to the
> Catalog
> > > > > > > interface.
> > > > > > > > > > > > > > > > > > This method can convert a Catalog
> instance
> > > to a
> > > > > > > > > Map<String,
> > > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > properties, and the default
> implementation
> > > will
> > > > > > throw
> > > > > > > > an
> > > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > > * Returns a map containing the
> properties
> > > of
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > > * @return a map containing the
> properties
> > > of
> > > > > the
> > > > > > > > > catalog
> > > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > > * @throws
> UnsupportedOperationException if
> > > > the
> > > > > > > > > > > implementing
> > > > > > > > > > > > > > class
> > > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > > * the default implementation of this
> > > method
> > > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > > default Map<String, String>
> toProperties()
> > > {
> > > > > > > > > > > > > > > > > > throw new
> > > > > UnsupportedOperationException("Please
> > > > > > > > > > implement
> > > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. If the user has configured a
> > > CatalogStore,
> > > > > the
> > > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > will be called when registering a Catalog
> > > > > instance
> > > > > > > with
> > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> Catalog
> > > > > > catalog).
> > > > > > > > The
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > will be converted to a Map<String,
> String>
> > > > > > properties
> > > > > > > > and
> > > > > > > > > > > saved
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances
> do
> > > not
> > > > > > > > implement
> > > > > > > > > > this
> > > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. If the user does not use a
> CatalogStore,
> > > > the
> > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > will not be called, ensuring consistency
> with
> > > > the
> > > > > > > > > original
> > > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > > > catalogs
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > > we can start from the Catalog itself and
> > > > provide
> > > > > a
> > > > > > > > > > dedicated
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > public class ConfigurationCatalog
> implements
> > > > > > Catalog
> > > > > > > {
> > > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > > > > properties)
> > > > > > > {
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon
> FY <
> > > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have no questions about the
> functionality
> > > > of
> > > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > > `registerCatalog`
> > > > > > > or
> > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `CatalogManager`. The implementation in
> > > Trino
> > > > > is
> > > > > > > also
> > > > > > > > > the
> > > > > > > > > > > > same,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > > > > `createCatalog`,
> > > > > > > > > > > which
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > > > `registerCatalog`
> > > > > > > > > or
> > > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > > > > catalogs`
> > > > > > > or
> > > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned
> before,
> > > the
> > > > > > > > > `Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs` is a cache in
> `CatalogManager`.
> > > How
> > > > > > about
> > > > > > > > > > saving
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > > `CatalogStore
> > > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng
> Jin <
> > > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you for your reply. I am very
> sorry
> > > > for
> > > > > > the
> > > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > > discussion.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> [1]
> > > for
> > > > > > this
> > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > Yes, there was indeed such a
> discussion
> > > > > before.
> > > > > > > > > > However,
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > whole solution, I found that the
> logic of
> > > > > > > > > > CatalogManager
> > > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > > need to change much. *We cannot only
> > > > persist
> > > > > > > > Catalog
> > > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > > so exposing only
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > not be enough to save Catalogs,
> because
> > > in
> > > > > the
> > > > > > > end
> > > > > > > > we
> > > > > > > > > > > still
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > > the configurations corresponding to
> the
> > > > > Catalog
> > > > > > > > > > > instances.
> > > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > > > > interface
> > > > > > > for
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > persistence. Regarding this part, I
> also
> > > > took
> > > > > > > > > > inspiration
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> catalog
> > > with
> > > > > the
> > > > > > > > same
> > > > > > > > > > name
> > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> > > > provide
> > > > > a
> > > > > > > > > detailed
> > > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > code examples for this part, and add
> it
> > > to
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> `addCatalog`
> > > > still
> > > > > > > looks
> > > > > > > > > > > confused
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > > CatalogStore,
> > > > > > > > > how
> > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > > opinion,
> > > > > > > you
> > > > > > > > > can
> > > > > > > > > > > > decide
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to
> me.
> > > I
> > > > > > don't
> > > > > > > > see
> > > > > > > > > > any
> > > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement,
> which
> > > > > method
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> I'm
> > > > > > wondering
> > > > > > > > > > whether
> > > > > > > > > > > > user
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original
> design
> > > was
> > > > to
> > > > > > add
> > > > > > > > it
> > > > > > > > > > > > directly
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt
> the
> > > > > > existing
> > > > > > > > > > logic.
> > > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > we can do both: save the
> configuration to
> > > > the
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to
> provide a
> > > > > > default
> > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > I think it is necessary, otherwise we
> > > would
> > > > > > need
> > > > > > > to
> > > > > > > > > > > > introduce
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > > Map to store the configuration for
> lazy
> > > > > > loading.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration
> for
> > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > made into an asynchronous interface
> > > because
> > > > > > > Catalog
> > > > > > > > > is
> > > > > > > > > > > > > already
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> initialization
> > > > for
> > > > > > > > catalog
> > > > > > > > > > than
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > Indeed, my description was
> inaccurate. It
> > > > > > should
> > > > > > > be
> > > > > > > > > > lazy
> > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > instead of asynchronous
> registration. I
> > > > have
> > > > > > > > already
> > > > > > > > > > > > updated
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM
> Shammon
> > > FY <
> > > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think if there is a
> `registerCatalog`
> > > > > > method
> > > > > > > in
> > > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > confuse users whether a method
> named
> > > > > > > `addCatalog`
> > > > > > > > > or
> > > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> > > catalog
> > > > > is a
> > > > > > > > > > `cache`,
> > > > > > > > > > > I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> > > > users.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion
> [1]
> > > for
> > > > > > this
> > > > > > > > > FLIP.
> > > > > > > > > > > > Please
> > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of
> that
> > > > > > > > discussion
> > > > > > > > > is
> > > > > > > > > > > to
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface
> and
> > > > > > implement
> > > > > > > it
> > > > > > > > > for
> > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > as memory, file and external
> system.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think there is a gap between the
> > > > current
> > > > > > FLIP
> > > > > > > > > > design
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > > discussion
> > > > > in
> > > > > > > > thread
> > > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM
> yuxia <
> > > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this
> FLIP.
> > > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling
> catalog
> > > > with
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > > name
> > > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> > > better
> > > > to
> > > > > > > > explain
> > > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > these methods and give a brief
> > > example
> > > > in
> > > > > > > this
> > > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name
> `addCatalog`
> > > > > still
> > > > > > > > looks
> > > > > > > > > > > > confused
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > > this method is for storing
> catalog to
> > > > > > > > > CatalogStore,
> > > > > > > > > > > how
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very
> personal
> > > > > > > opinion,
> > > > > > > > > you
> > > > > > > > > > > can
> > > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement,
> > > which
> > > > > > method
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > > `registerCatalog` or
> `addCatalog`?
> > > I'm
> > > > > > > > wondering
> > > > > > > > > > > > whether
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > > catalog to store with SQL
> stement.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > > provide a
> > > > > > > default
> > > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 4: About asynchronous
> registration
> > > for
> > > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > When creating a catalog with
> CREATE
> > > > > > > CATALOG,
> > > > > > > > > the
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > default.
> > > > > > > > > > > > > > > > > > > > > > If asynchronous registration is
> the
> > > > > default
> > > > > > > > > > behavior,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > > that user can switch to
> synchronous
> > > > > > > > registration
> > > > > > > > > > just
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > > > > `registerCatalog`
> > > > > > > > > > > be
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous
> registration,
> > > it
> > > > > may
> > > > > > > well
> > > > > > > > > > that
> > > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > > executes successfully, but then
> the
> > > > > > following
> > > > > > > > > > CREATE
> > > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to
> open.
> > > > > > > > > > > > > > > > > > > > > > I think it's a break change which
> > > > should
> > > > > be
> > > > > > > > > > > highlighted
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > > registration, I
> > > > > > > > would
> > > > > > > > > > > like
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > > will be an executor to open or
> > > register
> > > > > > > catalog
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > > initialized
> > > > > > if
> > > > > > > it
> > > > > > > > > has
> > > > > > > > > > > not
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > > yet. If the initialization
> process
> > > > fails,
> > > > > > > these
> > > > > > > > > > > > > statements
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > > initialization
> > > > > for
> > > > > > > > > catalog
> > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > > jinfeng1995@gmail.com
> > > > >
> > > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <dev@flink.apache.org
> >
> > > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午
> 3:27:45
> > > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295:
> Support
> > > > > > > persistence
> > > > > > > > > of
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for
> your
> > > > > > interest
> > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > > How to handle a catalog with
> the
> > > same
> > > > > > name
> > > > > > > > that
> > > > > > > > > > > > exists
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I believe this is a crucial
> point.
> > > > Based
> > > > > on
> > > > > > > my
> > > > > > > > > > > current
> > > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> > > Catalog>
> > > > > > > > catalogs
> > > > > > > > > > will
> > > > > > > > > > > > > serve
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and
> have
> > > the
> > > > > > > highest
> > > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There are three methods that can
> > > affect
> > > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method puts the catalog
> instance
> > > > > into
> > > > > > > the
> > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > > catalogName)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > > > instance
> > > > > > > > > > > corresponding
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > > catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > > > corresponding
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog>
> catalogs. If
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > does
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > > configuration
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > > initialized
> > > > > > > > Catalog
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The following two methods only
> modify
> > > > the
> > > > > > > > > > > configuration
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > > > > Map<String,
> > > > > > > > > > String>
> > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This method saves the properties
> to
> > > the
> > > > > > > > > > catalogStore
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > > whether there is a catalogName
> with
> > > the
> > > > > > same
> > > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String
> catalogName)
> > > > > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > > > > configuration
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The following are possible
> conflict
> > > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. When the corresponding
> catalogName
> > > > > > already
> > > > > > > > > > exists
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > > Map<String,
> > > > > > > > Catalog>,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > succeed and be directly saved to
> the
> > > > > > > > Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. When the corresponding
> catalogName
> > > > > > already
> > > > > > > > > > exists
> > > > > > > > > > > in
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > > > > Catalog>,
> > > > > > > the
> > > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog)
> method
> > > > will
> > > > > > > fail.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 3. When the corresponding
> catalogName
> > > > > > already
> > > > > > > > > > exists
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > > addCatalog(String
> > > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > String> properties) method can
> > > directly
> > > > > > save
> > > > > > > > the
> > > > > > > > > > > > > properties
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > > getCatalog(String
> > > > > > > > > > catalogName)
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > > initialization
> > > > > > > > because
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > > catalog instance already exists
> in
> > > > > catalogs
> > > > > > > and
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > > unregisterCatalog(String
> > > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > remove the instance
> corresponding to
> > > > the
> > > > > > > > original
> > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it will confuse users
> that
> > > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in
> the
> > > > > > > > `Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > `registerCatalog(String
> catalogName,
> > > > > > > > Map<String,
> > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > > > confusion. I
> > > > > > > > > suggest
> > > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > > addCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > > String> properties), as
> previously
> > > > > > mentioned
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > > catalogName,Catalog
> > > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > > instance,
> > > > > > > adding
> > > > > > > > > the
> > > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > > parameter to the
> > > registerCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > > method may not necessarily
> result in
> > > > lazy
> > > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > > encryption
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think encryption is necessary,
> but
> > > > > > perhaps
> > > > > > > > the
> > > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > > CatalogStore
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > > data
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > > FileCatalogStore
> > > > > > > could
> > > > > > > > > > > encrypt
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > does
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > as
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > > implementation, which saves
> Catalog
> > > > > > > > > configurations
> > > > > > > > > > in
> > > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to
> implement
> > > > > > > asynchronous
> > > > > > > > > > > loading
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > would need to introduce
> additional
> > > > cache.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > What if asynchronous
> registration
> > > > > failed?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This is also a critical concern.
> > > When
> > > > > > > > executing
> > > > > > > > > > DDL,
> > > > > > > > > > > > > DQL,
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > > statements that reference a
> specific
> > > > > > Catalog,
> > > > > > > > the
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > > will be initialized if it has not
> > > been
> > > > > > > > > initialized
> > > > > > > > > > > yet.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > initialization process fails,
> these
> > > > > > > statements
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> > > catalogs
> > > > > and
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the CatalogManager all have the
> same
> > > > > > catalog
> > > > > > > > > name,
> > > > > > > > > > > but
> > > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > > referring
> > > > > to
> > > > > > > the
> > > > > > > > > > > previous
> > > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > key principle is that the Catalog
> > > that
> > > > > has
> > > > > > > been
> > > > > > > > > > most
> > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > > will be given priority for
> subsequent
> > > > > use.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> > > Xianxun
> > > > > Ye
> > > > > > <
> > > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip,
> I do
> > > > > > believe
> > > > > > > > this
> > > > > > > > > > > Flip
> > > > > > > > > > > > > > could
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > > for users. The firm I work for
> also
> > > > > > manages a
> > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> > > > because
> > > > > of
> > > > > > > > > > loading a
> > > > > > > > > > > > > > number
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the
> user's
> > > > > Flink
> > > > > > > SQL
> > > > > > > > > > > through
> > > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> > > > improve
> > > > > > the
> > > > > > > > > speed
> > > > > > > > > > of
> > > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > > After reading flip-295, I have
> some
> > > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> > > > CREATE
> > > > > > > > CATALOG,
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > > registration method is used by
> > > default.
> > > > > > What
> > > > > > > if
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > > asynchronous
> > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2. I also have the same
> question
> > > with
> > > > > > > > Shammon’s
> > > > > > > > > > > first
> > > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > > > > CatalogStore
> > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> > > > catalog
> > > > > > > name,
> > > > > > > > > but
> > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > different catalog instances. For
> > > > example:
> > > > > > > > > > catalog1->
> > > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 ->
> hiveCatalog
> > > > in
> > > > > > the
> > > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > > > `catalogs`
> > > > > > and
> > > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Feng,
Just naming, maybe `createCatalog` in TableEnv, I can see many
functions are converted to createxxx from registerxxx.
On Fri, Jun 2, 2023 at 11:04 AM Feng Jin <ji...@gmail.com> wrote:
>
> Hi jark, thanks for your suggestion.
>
> > 1. How to register the CatalogStore for Table API? I think the
> CatalogStore should be immutable once TableEnv is created. Otherwise, there
> might be some data inconsistencies when CatalogStore is changed.
>
> Yes, We should initialize the CatalogStore when creating the TableEnv.
> Therefore, my current proposal is to add a method to configure the
> CatalogStore in EnvironmentSettings.
>
> // Initialize a catalog Store
> CatalogStore catalogStore = new FileCatalogStore("");
>
> // set up the Table API
> final EnvironmentSettings settings =
> EnvironmentSettings.newInstance().inBatchMode()
> .withCatalogStore(catalogStore)
> .build();
>
> final TableEnvironment tableEnv = TableEnvironment.create(settings);
>
>
> > 2. Why does the CatalogStoreFactory interface only have a default method,
> not an interface method?
>
> Sorry, While I did refer to the Catalog interface, I agree that as a new
> interface, the CatalogStoreFactory should not have a default method but an
> interface method. I have already modified the interface.
>
> > 3. Please mention the alternative API in Javadoc for the
> deprecated`registerCatalog`.
> > 4. In the "Compatibility" section, would be better to mention the changed
> behavior of CREATE CATALOG statement if FileCatalogStore (or other
> persisted catalog store) is used.
>
> Thanks for the suggestion, I have updated the FLIP [1].
>
>
> [1].
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
>
>
> Best,
> Feng
>
> On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
>
> > Hi Feng,
> >
> > This is a useful FLIP. Thanks for starting this discussion.
> > The current design looks pretty good to me. I just have some minor
> > comments.
> >
> > 1. How to register the CatalogStore for Table API? I think the CatalogStore
> > should be immutable once TableEnv is created. Otherwise, there might be
> > some data inconsistencies when CatalogStore is changed.
> >
> > 2. Why does the CatalogStoreFactory interface only have a default method,
> > not an interface method?
> >
> > 3. Please mention the alternative API in Javadoc for the deprecated
> > `registerCatalog`.
> >
> > 4. In the "Compatibility" section, would be better to mention the changed
> > behavior of CREATE CATALOG statement if FileCatalogStore (or other
> > persisted catalog store) is used.
> >
> >
> > Best,
> > Jark
> >
> > On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> >
> > > Hi ， thanks all for reviewing the flip.
> > >
> > > @Ron
> > >
> > > > Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > need
> > > to provide a default implementation?
> > >
> > > Yes, we will provide a default InMemoryCatalogStoreFactory to create an
> > > InMemoryCatalogStore.
> > >
> > > > If we get a Catalog from CatalogStore, after initializing it, whether
> > we
> > > put it in Map<String, Catalog> catalogs again?
> > >
> > > Yes, in the current design, catalogs are stored as snapshots, and once
> > > initialized, the Catalog will be placed in the Map<String, Catalog>
> > > catalogs.
> > > Subsequently, the Map<String, Catalog> catalogs will be the primary
> > source
> > > for obtaining the corresponding Catalog.
> > >
> > > > how about renaming them to `catalog.store.type` and
> > > `catalog.store.path`?
> > >
> > > I think it is okay. Adding "sql" at the beginning may seem a bit
> > strange. I
> > > will update the FLIP.
> > >
> > >
> > >
> > > @Shammon
> > >
> > > Thank you for the review. I have made the necessary corrections.
> > > Regarding the modifications made to the Public Interface, I have also
> > > included the relevant changes to the `TableEnvironment`.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
> > >
> > > > Hi feng,
> > > >
> > > > Thanks for updating, I have some minor comments
> > > >
> > > > 1. The modification of `CatalogManager` should not be in `Public
> > > > Interfaces`, it is not a public interface.
> > > >
> > > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > > `CatalogStoreFactory`
> > > >
> > > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > > catalogStore.get(catalogName);`
> > > >
> > > > Best,
> > > > Shammon FY
> > > >
> > > >
> > > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> > > >
> > > > > Hi, Feng
> > > > >
> > > > > Thanks for driving this FLIP, this proposal is very useful for
> > catalog
> > > > > management.
> > > > > I have some small questions:
> > > > >
> > > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > > > need
> > > > > to provide a default implementation?
> > > > > 2. If we get Catalog from CatalogStore, after initializing it,
> > whether
> > > we
> > > > > put it to Map<String, Catalog> catalogs again?
> > > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > > `catalog.store.type` and `catalog.store.path`?
> > > > >
> > > > > Best,
> > > > > Ron
> > > > >
> > > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > > >
> > > > > > Hi yuxia
> > > > > >
> > > > > > > But from the code in Proposed Changes, once we register the
> > > Catalog,
> > > > > we
> > > > > > initialize it and open it. right?
> > > > > >
> > > > > > Yes, In order to avoid inconsistent semantics of the original
> > CREATE
> > > > > > CATALOG DDL, Catalog will be directly initialized in
> > registerCatalog
> > > so
> > > > > > that parameter validation can be performed.
> > > > > >
> > > > > > In the current design, lazy initialization is mainly reflected in
> > > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > > configurations,
> > > > > > only initialization is required in getCatalog.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Feng.
> > > > > > > I'm trying to understanding the meaning of *lazy initialization*.
> > > If
> > > > > i'm
> > > > > > > wrong, please correct me.
> > > > > > >
> > > > > > > IIUC, lazy initialization means only you need to access the
> > > catalog,
> > > > > then
> > > > > > > you initialize it. But from the code in Proposed Changes, once we
> > > > > > register
> > > > > > > the Catalog,
> > > > > > > we initialize it and open it. right?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Yuxia
> > > > > > >
> > > > > > > ----- 原始邮件 -----
> > > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > > configuration
> > > > > > > and asynchronous registration
> > > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for your effort! +1 for the proposal.
> > > > > > >
> > > > > > > One of the major changes is that current design will provide
> > > > > > > Map<String,Catalog> catalogs as a snapshot instead of a cache,
> > > which
> > > > > > means
> > > > > > > once it has been initialized, any changes done by other sessions
> > > will
> > > > > not
> > > > > > > affect it. Point 6 described follow-up options for further
> > > > improvement.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all, I would like to update you on the latest progress of
> > the
> > > > > FLIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai
> > > Fang
> > > > > > and I
> > > > > > > > had an offline discussion regarding the overall solution for
> > > Flink
> > > > > > > > CatalogStore. We have reached a consensus and I have updated
> > the
> > > > > final
> > > > > > > > solution in FLIP.
> > > > > > > >
> > > > > > > > Next, let me briefly describe the entire design:
> > > > > > > >
> > > > > > > > 1.
> > > > > > > >
> > > > > > > > Introduce CatalogDescriptor to store catalog configuration
> > > > similar
> > > > > > to
> > > > > > > > TableDescriptor.
> > > > > > > > 2.
> > > > > > > >
> > > > > > > > The two key functions of CatalogStore - void
> > > storeCatalog(String
> > > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > > getCatalog(String)
> > > > > > > > will both use CatalogDescriptor instead of Catalog instance.
> > > > This
> > > > > > way,
> > > > > > > > CatalogStore will only be responsible for saving and
> > > retrieving
> > > > > > > catalog
> > > > > > > > configurations without having to initialize catalogs.
> > > > > > > > 3.
> > > > > > > >
> > > > > > > > The default registerCatalog(String catalogName, Catalog
> > > catalog)
> > > > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > > > 4.
> > > > > > > >
> > > > > > > > A new function registerCatalog(String catalogName,
> > > > > CatalogDescriptor
> > > > > > > > catalog) will be added to serve as the default registration
> > > > > function
> > > > > > > for
> > > > > > > > catalogs in CatalogManager.
> > > > > > > > 5.
> > > > > > > >
> > > > > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > > > > unchanged
> > > > > > > and
> > > > > > > > save initialized catalogs.This means that deletion
> > operations
> > > > from
> > > > > > one
> > > > > > > > session won't synchronize with other sessions.
> > > > > > > > 6.
> > > > > > > >
> > > > > > > > To support multi-session synchronization scenarios for
> > > deletions
> > > > > > later
> > > > > > > > on we should make Map<String,Catalog>catalogs
> > > configurable.There
> > > > > may
> > > > > > > be
> > > > > > > > three possible situations:
> > > > > > > >
> > > > > > > > a.Default caching of all initialized catalogs
> > > > > > > >
> > > > > > > > b.Introduction of LRU cache logic which can automatically
> > > clear
> > > > > > > > long-unused catalogs.
> > > > > > > >
> > > > > > > > c.No caching of any instances; each call to getCatalog
> > > creates a
> > > > > new
> > > > > > > > instance.
> > > > > > > >
> > > > > > > >
> > > > > > > > This is the document for discussion:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > > >
> > > > > > > > This is the final proposal document:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > > >
> > > > > > > >
> > > > > > > > Thank you very much for your attention and suggestions on this
> > > > > FLIP. A
> > > > > > > > special thanks to Hang Ruan for his suggestions on the entire
> > > > design
> > > > > > and
> > > > > > > > organizing offline discussions.
> > > > > > > >
> > > > > > > > If you have any further suggestions or feedback about this FLIP
> > > > > please
> > > > > > > feel
> > > > > > > > free to share.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > Thanks for improving the FLIP. It looks good to me. We could
> > > > still
> > > > > > > > > reconsider in the future how to provide more common built-in
> > > > cache
> > > > > > > > > functionality in CatalogManager so that not every
> > CatalogSotre
> > > > > > > > > implementation has to take care of it.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> > jinfeng1995@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jing,
> > > > > > > > > >
> > > > > > > > > > Thanks for your reply.
> > > > > > > > > >
> > > > > > > > > > > There might be more such issues. I would suggest you
> > > > > completely
> > > > > > > walk
> > > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > > >
> > > > > > > > > > I am very sorry for my carelessness and at the same time, I
> > > > > greatly
> > > > > > > > > > appreciate your careful review. I have thoroughly checked
> > the
> > > > > > entire
> > > > > > > > FLIP
> > > > > > > > > > and made corrections to these issues.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > > > CatalogManager
> > > > > > > > > > could work without Optional CatalogStore being configured.
> > > > > > > > > >
> > > > > > > > > > Yes, in the original design, CatalogStore was not necessary
> > > > > because
> > > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > > > > catalog
> > > > > > > > > > instances.
> > > > > > > > > > However, this caused inconsistency issues. Therefore, I
> > > > modified
> > > > > > this
> > > > > > > > > part
> > > > > > > > > > of the design and removed Map<String, Catalog> catalogs
> > from
> > > > > > > > > > CatalogManager.
> > > > > > > > > > At the same time, InMemoryCatalog will serve as the
> > default
> > > > > > > > CatalogStore
> > > > > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> > > > been
> > > > > > > moved
> > > > > > > > to
> > > > > > > > > > Rejected Alternatives.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > There are still many places contain inconsistent content,
> > > > e.g.
> > > > > > > > > > >
> > > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > > > > method
> > > > > > is
> > > > > > > > > > doing.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > There might be more such issues. I would suggest you
> > > > completely
> > > > > > > walk
> > > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > > >
> > > > > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> > > > build
> > > > > > the
> > > > > > > > > cache
> > > > > > > > > > > functionality in the CatalogStore? This is a very
> > different
> > > > > > design
> > > > > > > > > > concept
> > > > > > > > > > > from what the current FLIP described. If I am not
> > mistaken,
> > > > > with
> > > > > > > the
> > > > > > > > > > > current FLIP design, CatalogManager could work without
> > > > Optional
> > > > > > > > > > > CatalogStore being configured. That is the reason why I
> > > > > mentioned
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > last email that the example code wrt the Optional
> > > > CatalogStore
> > > > > is
> > > > > > > not
> > > > > > > > > > > correct.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > > jinfeng1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you very much for the careful review. I have
> > > already
> > > > > made
> > > > > > > the
> > > > > > > > > > > > relevant changes.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > Speaking of the conflict issues in the
> > multi-instance
> > > > > > > > scenarios, I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > > >
> > > > > > > > > > > > Currently, there are conflicts in multiple scenarios
> > with
> > > > the
> > > > > > > > current
> > > > > > > > > > > > design. I am thinking whether we should remove
> > > 'Map<String,
> > > > > > > > Catalog>'
> > > > > > > > > > and
> > > > > > > > > > > > make Cache the default behavior of
> > InMemoryCatalogStore.
> > > > This
> > > > > > > way,
> > > > > > > > > > users
> > > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > > multi-instance
> > > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for working on the FLIP. There are still some
> > > NIT
> > > > > > issues
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > like:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used
> > as
> > > > > > > > > CatalogStore
> > > > > > > > > > > > > instead of Optional in the code example. It should be
> > > > fine
> > > > > to
> > > > > > > use
> > > > > > > > > it
> > > > > > > > > > as
> > > > > > > > > > > > > pseudo code for now and update it after you submit
> > the
> > > > PR.
> > > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > > > rejected
> > > > > > > > section
> > > > > > > > > > > which
> > > > > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > > > scenarios,
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > > > sure if this is the intended behaviour. If
> > Map<String,
> > > > > > Catalog>
> > > > > > > > > > > catalogs
> > > > > > > > > > > > is
> > > > > > > > > > > > > used as a cache, it should be invalid, once the
> > related
> > > > > > catalog
> > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > removed from the CatalogStore by another instance.
> > Did
> > > I
> > > > > miss
> > > > > > > > > > > something?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > > I think this is a good function name, I have
> > updated
> > > it
> > > > > in
> > > > > > > the
> > > > > > > > > > > > > > documentation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Some common cache features should be
> > implemented
> > > > > > > > > > > > > > Thank you for the suggestion. If alternative 1
> > turns
> > > > out
> > > > > to
> > > > > > > be
> > > > > > > > > more
> > > > > > > > > > > > > > appropriate later, I will improve this part of the
> > > > > content.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As the above discussion moves forward, the
> > option 2
> > > > > > > solution
> > > > > > > > > > looks
> > > > > > > > > > > > more
> > > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > > Yes, after discussing with Shammon offline, we
> > think
> > > > that
> > > > > > > > > solution
> > > > > > > > > > 2
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > be more suitable and also avoid any inconsistency
> > > > issues.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There are some inconsistent descriptions in the
> > > > > content.
> > > > > > > > Would
> > > > > > > > > > you
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > > I will do my best to improve the document and
> > > > appreciate
> > > > > > your
> > > > > > > > > > > > > suggestions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > > > > Alternatives`
> > > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> > Alternatives`.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best
> > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > > zjureel@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I found there are two options in `Proposed
> > > Changes`,
> > > > > can
> > > > > > > you
> > > > > > > > > put
> > > > > > > > > > > the
> > > > > > > > > > > > > > > unselected option in `Rejected Alternatives`? I
> > > think
> > > > > > this
> > > > > > > > may
> > > > > > > > > > help
> > > > > > > > > > > > us
> > > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> > confused
> > > > > after
> > > > > > > > > > > completely
> > > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > > > addCatalog()
> > > > > > > > have
> > > > > > > > > > no
> > > > > > > > > > > > big
> > > > > > > > > > > > > > > > difference based on their names. One of them is
> > > > > > > responsible
> > > > > > > > > for
> > > > > > > > > > > > data
> > > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > > > > catalogs
> > > > > > is
> > > > > > > > > used
> > > > > > > > > > > as a
> > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > and catalogStore is used for data persistence.
> > I
> > > > > would
> > > > > > > > > suggest
> > > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > > their purpose conceptually and clearly in the
> > > FLIP.
> > > > > > Some
> > > > > > > > > common
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > features should be implemented, i.e. data in
> > the
> > > > > cache
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > store
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > be consistent. Same Catalog instance should be
> > > > found
> > > > > in
> > > > > > > the
> > > > > > > > > > store
> > > > > > > > > > > > and
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the cache(either it has been initialized or it
> > > will
> > > > > be
> > > > > > > lazy
> > > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > > for the same catalog name. The consistency will
> > > be
> > > > > > taken
> > > > > > > > care
> > > > > > > > > > of
> > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > > 3. As the above discussion moves forward, the
> > > > option
> > > > > 2
> > > > > > > > > solution
> > > > > > > > > > > > looks
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > > > > issues
> > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > previously with option 1 are not solved yet. Do
> > > you
> > > > > > still
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > > both options and ask for suggestions for both
> > of
> > > > > them?
> > > > > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > > > > inconsistent
> > > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the content. Would you like to clean them up?
> > > > > Thanks!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for your response, and I completely
> > > > agree
> > > > > > > with
> > > > > > > > > your
> > > > > > > > > > > > point
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > > Initially, I may have over complicated the
> > > whole
> > > > > > issue.
> > > > > > > > > First
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > > we need to consider the persistence of the
> > > > > Catalog's
> > > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > > If we only need to provide persistence for
> > > > Catalog
> > > > > > > > > > > Configuration,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > > > > interface.
> > > > > > > > > > > > > > > > > This method can convert a Catalog instance
> > to a
> > > > > > > > Map<String,
> > > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > properties, and the default implementation
> > will
> > > > > throw
> > > > > > > an
> > > > > > > > > > > > exception.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > > * Returns a map containing the properties
> > of
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > > object.
> > > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > > * @return a map containing the properties
> > of
> > > > the
> > > > > > > > catalog
> > > > > > > > > > > > object
> > > > > > > > > > > > > > > > > * @throws UnsupportedOperationException if
> > > the
> > > > > > > > > > implementing
> > > > > > > > > > > > > class
> > > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > > * the default implementation of this
> > method
> > > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > > default Map<String, String> toProperties()
> > {
> > > > > > > > > > > > > > > > > throw new
> > > > UnsupportedOperationException("Please
> > > > > > > > > implement
> > > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. If the user has configured a
> > CatalogStore,
> > > > the
> > > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > will be called when registering a Catalog
> > > > instance
> > > > > > with
> > > > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > > > catalog).
> > > > > > > The
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > will be converted to a Map<String, String>
> > > > > properties
> > > > > > > and
> > > > > > > > > > saved
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances do
> > not
> > > > > > > implement
> > > > > > > > > this
> > > > > > > > > > > > > method,
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. If the user does not use a CatalogStore,
> > > the
> > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > will not be called, ensuring consistency with
> > > the
> > > > > > > > original
> > > > > > > > > > > > process.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > > catalogs
> > > > > and
> > > > > > > the
> > > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > > we can start from the Catalog itself and
> > > provide
> > > > a
> > > > > > > > > dedicated
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > > > > Catalog
> > > > > > {
> > > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > > > properties)
> > > > > > {
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have no questions about the functionality
> > > of
> > > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > different operations of multiple
> > > > > `registerCatalog`
> > > > > > or
> > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > `CatalogManager`. The implementation in
> > Trino
> > > > is
> > > > > > also
> > > > > > > > the
> > > > > > > > > > > same,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > > > `createCatalog`,
> > > > > > > > > > which
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > > `registerCatalog`
> > > > > > > > or
> > > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > > > catalogs`
> > > > > > or
> > > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > > catalogStore`. As you mentioned before,
> > the
> > > > > > > > `Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`.
> > How
> > > > > about
> > > > > > > > > saving
> > > > > > > > > > > the
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > > `CatalogStore
> > > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > > when it is registered or added in
> > > > > `CatalogManager`?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry
> > > for
> > > > > the
> > > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > > by my deviation from the original
> > > discussion.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> > for
> > > > > this
> > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> > > > before.
> > > > > > > > > However,
> > > > > > > > > > > in
> > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > > > > CatalogManager
> > > > > > > > > > > > itself
> > > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > > need to change much. *We cannot only
> > > persist
> > > > > > > Catalog
> > > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > not be enough to save Catalogs, because
> > in
> > > > the
> > > > > > end
> > > > > > > we
> > > > > > > > > > still
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > > the configurations corresponding to the
> > > > Catalog
> > > > > > > > > > instances.
> > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > > > interface
> > > > > > for
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > persistence. Regarding this part, I also
> > > took
> > > > > > > > > inspiration
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> > with
> > > > the
> > > > > > > same
> > > > > > > > > name
> > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> > > provide
> > > > a
> > > > > > > > detailed
> > > > > > > > > > > > > > description
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > code examples for this part, and add it
> > to
> > > > the
> > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > > still
> > > > > > looks
> > > > > > > > > > confused
> > > > > > > > > > > > to
> > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > CatalogStore,
> > > > > > > > how
> > > > > > > > > > > about
> > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > opinion,
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > decide
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me.
> > I
> > > > > don't
> > > > > > > see
> > > > > > > > > any
> > > > > > > > > > > > > problem
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> > > > method
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > > wondering
> > > > > > > > > whether
> > > > > > > > > > > user
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original design
> > was
> > > to
> > > > > add
> > > > > > > it
> > > > > > > > > > > directly
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > > > > existing
> > > > > > > > > logic.
> > > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > we can do both: save the configuration to
> > > the
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > > > > default
> > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > I think it is necessary, otherwise we
> > would
> > > > > need
> > > > > > to
> > > > > > > > > > > introduce
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > > > > loading.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > > > catalog.
> > > > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > made into an asynchronous interface
> > because
> > > > > > Catalog
> > > > > > > > is
> > > > > > > > > > > > already
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> > > for
> > > > > > > catalog
> > > > > > > > > than
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > > > > should
> > > > > > be
> > > > > > > > > lazy
> > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > instead of asynchronous registration. I
> > > have
> > > > > > > already
> > > > > > > > > > > updated
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon
> > FY <
> > > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > > > > method
> > > > > > in
> > > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > confuse users whether a method named
> > > > > > `addCatalog`
> > > > > > > > or
> > > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> > catalog
> > > > is a
> > > > > > > > > `cache`,
> > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> > > users.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> > for
> > > > > this
> > > > > > > > FLIP.
> > > > > > > > > > > Please
> > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > > > > discussion
> > > > > > > > is
> > > > > > > > > > to
> > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > > > > implement
> > > > > > it
> > > > > > > > for
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think there is a gap between the
> > > current
> > > > > FLIP
> > > > > > > > > design
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > > What about the proposal of the
> > discussion
> > > > in
> > > > > > > thread
> > > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > >
> > > > > > > > > >
> > > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> > > with
> > > > > the
> > > > > > > > same
> > > > > > > > > > name
> > > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> > better
> > > to
> > > > > > > explain
> > > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > these methods and give a brief
> > example
> > > in
> > > > > > this
> > > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > > > still
> > > > > > > looks
> > > > > > > > > > > confused
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > > > CatalogStore,
> > > > > > > > > > how
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > > > opinion,
> > > > > > > > you
> > > > > > > > > > can
> > > > > > > > > > > > > decide
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement,
> > which
> > > > > method
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> > I'm
> > > > > > > wondering
> > > > > > > > > > > whether
> > > > > > > > > > > > > user
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> > provide a
> > > > > > default
> > > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration
> > for
> > > > > > catalog.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > > > > CATALOG,
> > > > > > > > the
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration method is used by
> > default.
> > > > > > > > > > > > > > > > > > > > > If asynchronous registration is the
> > > > default
> > > > > > > > > behavior,
> > > > > > > > > > > it
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > > > > registration
> > > > > > > > > just
> > > > > > > > > > > > like
> > > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > > > `registerCatalog`
> > > > > > > > > > be
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration,
> > it
> > > > may
> > > > > > well
> > > > > > > > > that
> > > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > > executes successfully, but then the
> > > > > following
> > > > > > > > > CREATE
> > > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > > > > I think it's a break change which
> > > should
> > > > be
> > > > > > > > > > highlighted
> > > > > > > > > > > > in
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > > registration, I
> > > > > > > would
> > > > > > > > > > like
> > > > > > > > > > > to
> > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > > will be an executor to open or
> > register
> > > > > > catalog
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > > initialized
> > > > > if
> > > > > > it
> > > > > > > > has
> > > > > > > > > > not
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > > yet. If the initialization process
> > > fails,
> > > > > > these
> > > > > > > > > > > > statements
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > > It looks more like lazy
> > initialization
> > > > for
> > > > > > > > catalog
> > > > > > > > > > than
> > > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> > jinfeng1995@gmail.com
> > > >
> > > > > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > > > > persistence
> > > > > > > > of
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > > > > interest
> > > > > > in
> > > > > > > > > this
> > > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > > How to handle a catalog with the
> > same
> > > > > name
> > > > > > > that
> > > > > > > > > > > exists
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I believe this is a crucial point.
> > > Based
> > > > on
> > > > > > my
> > > > > > > > > > current
> > > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> > Catalog>
> > > > > > > catalogs
> > > > > > > > > will
> > > > > > > > > > > > serve
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > > for instantiated catalogs and have
> > the
> > > > > > highest
> > > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There are three methods that can
> > affect
> > > > the
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> > catalogName,
> > > > > > Catalog
> > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method puts the catalog instance
> > > > into
> > > > > > the
> > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > > catalogName)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > > instance
> > > > > > > > > > corresponding
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> > catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > > corresponding
> > > > > > > > > catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If
> > > the
> > > > > > > catalog
> > > > > > > > > > does
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > > configuration
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > > initialized
> > > > > > > Catalog
> > > > > > > > > > > instance
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The following two methods only modify
> > > the
> > > > > > > > > > configuration
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > > > Map<String,
> > > > > > > > > String>
> > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This method saves the properties to
> > the
> > > > > > > > > catalogStore
> > > > > > > > > > > and
> > > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > > whether there is a catalogName with
> > the
> > > > > same
> > > > > > > > name.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > > > configuration
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The following are possible conflict
> > > > > > scenarios:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > > > > already
> > > > > > > > > exists
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> > Map<String,
> > > > > > > Catalog>,
> > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > method
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > > > > Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > > > > already
> > > > > > > > > exists
> > > > > > > > > > in
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > > > Catalog>,
> > > > > > the
> > > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method
> > > will
> > > > > > fail.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > > > > already
> > > > > > > > > exists
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > > addCatalog(String
> > > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > String> properties) method can
> > directly
> > > > > save
> > > > > > > the
> > > > > > > > > > > > properties
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalogStore, but the
> > getCatalog(String
> > > > > > > > > catalogName)
> > > > > > > > > > > > method
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > use the new properties for
> > > initialization
> > > > > > > because
> > > > > > > > > the
> > > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > > catalog instance already exists in
> > > > catalogs
> > > > > > and
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > > unregisterCatalog(String
> > > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > remove the instance corresponding to
> > > the
> > > > > > > original
> > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > > > > `Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > > > > Map<String,
> > > > > > > > > > > String>
> > > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > > confusion. I
> > > > > > > > suggest
> > > > > > > > > > > > > changing
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > > addCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > > String> properties), as previously
> > > > > mentioned
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > > catalogName,Catalog
> > > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> > instance,
> > > > > > adding
> > > > > > > > the
> > > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > > parameter to the
> > registerCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > > method may not necessarily result in
> > > lazy
> > > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Do we need to think about
> > encryption
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > > > > perhaps
> > > > > > > the
> > > > > > > > > > > > encryption
> > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > > should be implemented in the
> > > CatalogStore
> > > > > > when
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > > FileCatalogStore
> > > > > > could
> > > > > > > > > > encrypt
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > > > MemoryCatalogStore
> > > > > > > > > > > does
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > > > MemoryCatalogStore
> > > > > > > > > > as
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > > > > configurations
> > > > > > > > > in
> > > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > > > > asynchronous
> > > > > > > > > > loading
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > would need to introduce additional
> > > cache.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > What if asynchronous registration
> > > > failed?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This is also a critical concern.
> > When
> > > > > > > executing
> > > > > > > > > DDL,
> > > > > > > > > > > > DQL,
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > > statements that reference a specific
> > > > > Catalog,
> > > > > > > the
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > > will be initialized if it has not
> > been
> > > > > > > > initialized
> > > > > > > > > > yet.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > initialization process fails, these
> > > > > > statements
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> > catalogs
> > > > and
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > > > > catalog
> > > > > > > > name,
> > > > > > > > > > but
> > > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> > referring
> > > > to
> > > > > > the
> > > > > > > > > > previous
> > > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > key principle is that the Catalog
> > that
> > > > has
> > > > > > been
> > > > > > > > > most
> > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > > will be given priority for subsequent
> > > > use.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> > Xianxun
> > > > Ye
> > > > > <
> > > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > > > > believe
> > > > > > > this
> > > > > > > > > > Flip
> > > > > > > > > > > > > could
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > > > > manages a
> > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> > > because
> > > > of
> > > > > > > > > loading a
> > > > > > > > > > > > > number
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> > > > Flink
> > > > > > SQL
> > > > > > > > > > through
> > > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> > > improve
> > > > > the
> > > > > > > > speed
> > > > > > > > > of
> > > > > > > > > > > > task
> > > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > > > > questions:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> > > CREATE
> > > > > > > CATALOG,
> > > > > > > > > the
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > > registration method is used by
> > default.
> > > > > What
> > > > > > if
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > > asynchronous
> > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2. I also have the same question
> > with
> > > > > > > Shammon’s
> > > > > > > > > > first
> > > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > > > CatalogStore
> > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> > > catalog
> > > > > > name,
> > > > > > > > but
> > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > different catalog instances. For
> > > example:
> > > > > > > > > catalog1->
> > > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog
> > > in
> > > > > the
> > > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > > `catalogs`
> > > > > and
> > > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Feng Jin <ji...@gmail.com>.
Hi jark, thanks for your suggestion.
> 1. How to register the CatalogStore for Table API? I think the
CatalogStore should be immutable once TableEnv is created. Otherwise, there
might be some data inconsistencies when CatalogStore is changed.
Yes, We should initialize the CatalogStore when creating the TableEnv.
Therefore, my current proposal is to add a method to configure the
CatalogStore in EnvironmentSettings.
// Initialize a catalog Store
CatalogStore catalogStore = new FileCatalogStore("");
// set up the Table API
final EnvironmentSettings settings =
EnvironmentSettings.newInstance().inBatchMode()
.withCatalogStore(catalogStore)
.build();
final TableEnvironment tableEnv = TableEnvironment.create(settings);
> 2. Why does the CatalogStoreFactory interface only have a default method,
not an interface method?
Sorry, While I did refer to the Catalog interface, I agree that as a new
interface, the CatalogStoreFactory should not have a default method but an
interface method. I have already modified the interface.
> 3. Please mention the alternative API in Javadoc for the
deprecated`registerCatalog`.
> 4. In the "Compatibility" section, would be better to mention the changed
behavior of CREATE CATALOG statement if FileCatalogStore (or other
persisted catalog store) is used.
Thanks for the suggestion, I have updated the FLIP [1].
[1].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
Best,
Feng
On Thu, Jun 1, 2023 at 9:22 PM Jark Wu <im...@gmail.com> wrote:
> Hi Feng,
>
> This is a useful FLIP. Thanks for starting this discussion.
> The current design looks pretty good to me. I just have some minor
> comments.
>
> 1. How to register the CatalogStore for Table API? I think the CatalogStore
> should be immutable once TableEnv is created. Otherwise, there might be
> some data inconsistencies when CatalogStore is changed.
>
> 2. Why does the CatalogStoreFactory interface only have a default method,
> not an interface method?
>
> 3. Please mention the alternative API in Javadoc for the deprecated
> `registerCatalog`.
>
> 4. In the "Compatibility" section, would be better to mention the changed
> behavior of CREATE CATALOG statement if FileCatalogStore (or other
> persisted catalog store) is used.
>
>
> Best,
> Jark
>
> On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
>
> > Hi ， thanks all for reviewing the flip.
> >
> > @Ron
> >
> > > Regarding the CatalogStoreFactory#createCatalogStore method, do we
> need
> > to provide a default implementation?
> >
> > Yes, we will provide a default InMemoryCatalogStoreFactory to create an
> > InMemoryCatalogStore.
> >
> > > If we get a Catalog from CatalogStore, after initializing it, whether
> we
> > put it in Map<String, Catalog> catalogs again?
> >
> > Yes, in the current design, catalogs are stored as snapshots, and once
> > initialized, the Catalog will be placed in the Map<String, Catalog>
> > catalogs.
> > Subsequently, the Map<String, Catalog> catalogs will be the primary
> source
> > for obtaining the corresponding Catalog.
> >
> > > how about renaming them to `catalog.store.type` and
> > `catalog.store.path`?
> >
> > I think it is okay. Adding "sql" at the beginning may seem a bit
> strange. I
> > will update the FLIP.
> >
> >
> >
> > @Shammon
> >
> > Thank you for the review. I have made the necessary corrections.
> > Regarding the modifications made to the Public Interface, I have also
> > included the relevant changes to the `TableEnvironment`.
> >
> >
> > Best,
> > Feng
> >
> >
> > On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
> >
> > > Hi feng,
> > >
> > > Thanks for updating, I have some minor comments
> > >
> > > 1. The modification of `CatalogManager` should not be in `Public
> > > Interfaces`, it is not a public interface.
> > >
> > > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > > `CatalogStoreFactory`
> > >
> > > 3. The code `Optional<Catalog> optionalDescriptor =
> > > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > > `Optional<CatalogDescriptor> optionalDescriptor =
> > > catalogStore.get(catalogName);`
> > >
> > > Best,
> > > Shammon FY
> > >
> > >
> > > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> > >
> > > > Hi, Feng
> > > >
> > > > Thanks for driving this FLIP, this proposal is very useful for
> catalog
> > > > management.
> > > > I have some small questions:
> > > >
> > > > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > > need
> > > > to provide a default implementation?
> > > > 2. If we get Catalog from CatalogStore, after initializing it,
> whether
> > we
> > > > put it to Map<String, Catalog> catalogs again?
> > > > 3. Regarding the options `sql.catalog.store.type` and
> > > > `sql.catalog.store.file.path`, how about renaming them to
> > > > `catalog.store.type` and `catalog.store.path`?
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > > >
> > > > > Hi yuxia
> > > > >
> > > > > > But from the code in Proposed Changes, once we register the
> > Catalog,
> > > > we
> > > > > initialize it and open it. right?
> > > > >
> > > > > Yes, In order to avoid inconsistent semantics of the original
> CREATE
> > > > > CATALOG DDL, Catalog will be directly initialized in
> registerCatalog
> > so
> > > > > that parameter validation can be performed.
> > > > >
> > > > > In the current design, lazy initialization is mainly reflected in
> > > > > getCatalog. If CatalogStore has already saved some catalog
> > > > configurations,
> > > > > only initialization is required in getCatalog.
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Mon, May 29, 2023 at 8:27 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> >
> > > > wrote:
> > > > >
> > > > > > Hi, Feng.
> > > > > > I'm trying to understanding the meaning of *lazy initialization*.
> > If
> > > > i'm
> > > > > > wrong, please correct me.
> > > > > >
> > > > > > IIUC, lazy initialization means only you need to access the
> > catalog,
> > > > then
> > > > > > you initialize it. But from the code in Proposed Changes, once we
> > > > > register
> > > > > > the Catalog,
> > > > > > we initialize it and open it. right?
> > > > > >
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > >
> > > > > > ----- 原始邮件 -----
> > > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > > configuration
> > > > > > and asynchronous registration
> > > > > >
> > > > > > Hi Feng,
> > > > > >
> > > > > > Thanks for your effort! +1 for the proposal.
> > > > > >
> > > > > > One of the major changes is that current design will provide
> > > > > > Map<String,Catalog> catalogs as a snapshot instead of a cache,
> > which
> > > > > means
> > > > > > once it has been initialized, any changes done by other sessions
> > will
> > > > not
> > > > > > affect it. Point 6 described follow-up options for further
> > > improvement.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi all, I would like to update you on the latest progress of
> the
> > > > FLIP.
> > > > > > >
> > > > > > >
> > > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai
> > Fang
> > > > > and I
> > > > > > > had an offline discussion regarding the overall solution for
> > Flink
> > > > > > > CatalogStore. We have reached a consensus and I have updated
> the
> > > > final
> > > > > > > solution in FLIP.
> > > > > > >
> > > > > > > Next, let me briefly describe the entire design:
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Introduce CatalogDescriptor to store catalog configuration
> > > similar
> > > > > to
> > > > > > > TableDescriptor.
> > > > > > > 2.
> > > > > > >
> > > > > > > The two key functions of CatalogStore - void
> > storeCatalog(String
> > > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > > getCatalog(String)
> > > > > > > will both use CatalogDescriptor instead of Catalog instance.
> > > This
> > > > > way,
> > > > > > > CatalogStore will only be responsible for saving and
> > retrieving
> > > > > > catalog
> > > > > > > configurations without having to initialize catalogs.
> > > > > > > 3.
> > > > > > >
> > > > > > > The default registerCatalog(String catalogName, Catalog
> > catalog)
> > > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > > 4.
> > > > > > >
> > > > > > > A new function registerCatalog(String catalogName,
> > > > CatalogDescriptor
> > > > > > > catalog) will be added to serve as the default registration
> > > > function
> > > > > > for
> > > > > > > catalogs in CatalogManager.
> > > > > > > 5.
> > > > > > >
> > > > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > > > unchanged
> > > > > > and
> > > > > > > save initialized catalogs.This means that deletion
> operations
> > > from
> > > > > one
> > > > > > > session won't synchronize with other sessions.
> > > > > > > 6.
> > > > > > >
> > > > > > > To support multi-session synchronization scenarios for
> > deletions
> > > > > later
> > > > > > > on we should make Map<String,Catalog>catalogs
> > configurable.There
> > > > may
> > > > > > be
> > > > > > > three possible situations:
> > > > > > >
> > > > > > > a.Default caching of all initialized catalogs
> > > > > > >
> > > > > > > b.Introduction of LRU cache logic which can automatically
> > clear
> > > > > > > long-unused catalogs.
> > > > > > >
> > > > > > > c.No caching of any instances; each call to getCatalog
> > creates a
> > > > new
> > > > > > > instance.
> > > > > > >
> > > > > > >
> > > > > > > This is the document for discussion:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > > >
> > > > > > > This is the final proposal document:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > > >
> > > > > > >
> > > > > > > Thank you very much for your attention and suggestions on this
> > > > FLIP. A
> > > > > > > special thanks to Hang Ruan for his suggestions on the entire
> > > design
> > > > > and
> > > > > > > organizing offline discussions.
> > > > > > >
> > > > > > > If you have any further suggestions or feedback about this FLIP
> > > > please
> > > > > > feel
> > > > > > > free to share.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Feng
> > > > > > >
> > > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >
> > > > > > > > Thanks for improving the FLIP. It looks good to me. We could
> > > still
> > > > > > > > reconsider in the future how to provide more common built-in
> > > cache
> > > > > > > > functionality in CatalogManager so that not every
> CatalogSotre
> > > > > > > > implementation has to take care of it.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <
> jinfeng1995@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jing,
> > > > > > > > >
> > > > > > > > > Thanks for your reply.
> > > > > > > > >
> > > > > > > > > > There might be more such issues. I would suggest you
> > > > completely
> > > > > > walk
> > > > > > > > > through the FLIP again and fix those issues
> > > > > > > > >
> > > > > > > > > I am very sorry for my carelessness and at the same time, I
> > > > greatly
> > > > > > > > > appreciate your careful review. I have thoroughly checked
> the
> > > > > entire
> > > > > > > FLIP
> > > > > > > > > and made corrections to these issues.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > > CatalogManager
> > > > > > > > > could work without Optional CatalogStore being configured.
> > > > > > > > >
> > > > > > > > > Yes, in the original design, CatalogStore was not necessary
> > > > because
> > > > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > > > catalog
> > > > > > > > > instances.
> > > > > > > > > However, this caused inconsistency issues. Therefore, I
> > > modified
> > > > > this
> > > > > > > > part
> > > > > > > > > of the design and removed Map<String, Catalog> catalogs
> from
> > > > > > > > > CatalogManager.
> > > > > > > > > At the same time, InMemoryCatalog will serve as the
> default
> > > > > > > CatalogStore
> > > > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> > > been
> > > > > > moved
> > > > > > > to
> > > > > > > > > Rejected Alternatives.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng
> > > > > > > > >
> > > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Feng,
> > > > > > > > > >
> > > > > > > > > > There are still many places contain inconsistent content,
> > > e.g.
> > > > > > > > > >
> > > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > > > method
> > > > > is
> > > > > > > > > doing.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > There might be more such issues. I would suggest you
> > > completely
> > > > > > walk
> > > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > > >
> > > > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> > > build
> > > > > the
> > > > > > > > cache
> > > > > > > > > > functionality in the CatalogStore? This is a very
> different
> > > > > design
> > > > > > > > > concept
> > > > > > > > > > from what the current FLIP described. If I am not
> mistaken,
> > > > with
> > > > > > the
> > > > > > > > > > current FLIP design, CatalogManager could work without
> > > Optional
> > > > > > > > > > CatalogStore being configured. That is the reason why I
> > > > mentioned
> > > > > > in
> > > > > > > > the
> > > > > > > > > > last email that the example code wrt the Optional
> > > CatalogStore
> > > > is
> > > > > > not
> > > > > > > > > > correct.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > > jinfeng1995@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jing
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > > >
> > > > > > > > > > > Thank you very much for the careful review. I have
> > already
> > > > made
> > > > > > the
> > > > > > > > > > > relevant changes.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > Speaking of the conflict issues in the
> multi-instance
> > > > > > > scenarios, I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > > >
> > > > > > > > > > > Currently, there are conflicts in multiple scenarios
> with
> > > the
> > > > > > > current
> > > > > > > > > > > design. I am thinking whether we should remove
> > 'Map<String,
> > > > > > > Catalog>'
> > > > > > > > > and
> > > > > > > > > > > make Cache the default behavior of
> InMemoryCatalogStore.
> > > This
> > > > > > way,
> > > > > > > > > users
> > > > > > > > > > > can implement their own CatalogStore to achieve
> > > > multi-instance
> > > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Feng
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for working on the FLIP. There are still some
> > NIT
> > > > > issues
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > like:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used
> as
> > > > > > > > CatalogStore
> > > > > > > > > > > > instead of Optional in the code example. It should be
> > > fine
> > > > to
> > > > > > use
> > > > > > > > it
> > > > > > > > > as
> > > > > > > > > > > > pseudo code for now and update it after you submit
> the
> > > PR.
> > > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > > rejected
> > > > > > > section
> > > > > > > > > > which
> > > > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > > > >
> > > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > > scenarios,
> > > > > > > I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > > > sure if this is the intended behaviour. If
> Map<String,
> > > > > Catalog>
> > > > > > > > > > catalogs
> > > > > > > > > > > is
> > > > > > > > > > > > used as a cache, it should be invalid, once the
> related
> > > > > catalog
> > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > removed from the CatalogStore by another instance.
> Did
> > I
> > > > miss
> > > > > > > > > > something?
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Jing
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > > jinfeng1995@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Jing
> > > > > > > > > > > > >
> > > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > > I think this is a good function name, I have
> updated
> > it
> > > > in
> > > > > > the
> > > > > > > > > > > > > documentation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Some common cache features should be
> implemented
> > > > > > > > > > > > > Thank you for the suggestion. If alternative 1
> turns
> > > out
> > > > to
> > > > > > be
> > > > > > > > more
> > > > > > > > > > > > > appropriate later, I will improve this part of the
> > > > content.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As the above discussion moves forward, the
> option 2
> > > > > > solution
> > > > > > > > > looks
> > > > > > > > > > > more
> > > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > > Yes, after discussing with Shammon offline, we
> think
> > > that
> > > > > > > > solution
> > > > > > > > > 2
> > > > > > > > > > > > might
> > > > > > > > > > > > > be more suitable and also avoid any inconsistency
> > > issues.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > There are some inconsistent descriptions in the
> > > > content.
> > > > > > > Would
> > > > > > > > > you
> > > > > > > > > > > > like
> > > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > > I will do my best to improve the document and
> > > appreciate
> > > > > your
> > > > > > > > > > > > suggestions.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > > > Alternatives`
> > > > > > > > > > > > > Sure, I have moved it to the `Rejected
> Alternatives`.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best
> > > > > > > > > > > > > Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > > zjureel@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I found there are two options in `Proposed
> > Changes`,
> > > > can
> > > > > > you
> > > > > > > > put
> > > > > > > > > > the
> > > > > > > > > > > > > > unselected option in `Rejected Alternatives`? I
> > think
> > > > > this
> > > > > > > may
> > > > > > > > > help
> > > > > > > > > > > us
> > > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for raising this FLIP. I am still
> confused
> > > > after
> > > > > > > > > > completely
> > > > > > > > > > > > > > reading
> > > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > > addCatalog()
> > > > > > > have
> > > > > > > > > no
> > > > > > > > > > > big
> > > > > > > > > > > > > > > difference based on their names. One of them is
> > > > > > responsible
> > > > > > > > for
> > > > > > > > > > > data
> > > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > > > catalogs
> > > > > is
> > > > > > > > used
> > > > > > > > > > as a
> > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > and catalogStore is used for data persistence.
> I
> > > > would
> > > > > > > > suggest
> > > > > > > > > > > > > describing
> > > > > > > > > > > > > > > their purpose conceptually and clearly in the
> > FLIP.
> > > > > Some
> > > > > > > > common
> > > > > > > > > > > cache
> > > > > > > > > > > > > > > features should be implemented, i.e. data in
> the
> > > > cache
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > store
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > be consistent. Same Catalog instance should be
> > > found
> > > > in
> > > > > > the
> > > > > > > > > store
> > > > > > > > > > > and
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the cache(either it has been initialized or it
> > will
> > > > be
> > > > > > lazy
> > > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > > for the same catalog name. The consistency will
> > be
> > > > > taken
> > > > > > > care
> > > > > > > > > of
> > > > > > > > > > > > while
> > > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > > 3. As the above discussion moves forward, the
> > > option
> > > > 2
> > > > > > > > solution
> > > > > > > > > > > looks
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > > > issues
> > > > > > > > > mentioned
> > > > > > > > > > > > > > > previously with option 1 are not solved yet. Do
> > you
> > > > > still
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > > propose
> > > > > > > > > > > > > > > both options and ask for suggestions for both
> of
> > > > them?
> > > > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > > > inconsistent
> > > > > > > > > > > > descriptions
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the content. Would you like to clean them up?
> > > > Thanks!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you for your response, and I completely
> > > agree
> > > > > > with
> > > > > > > > your
> > > > > > > > > > > point
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > > Initially, I may have over complicated the
> > whole
> > > > > issue.
> > > > > > > > First
> > > > > > > > > > and
> > > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > > we need to consider the persistence of the
> > > > Catalog's
> > > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > > If we only need to provide persistence for
> > > Catalog
> > > > > > > > > > Configuration,
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > > > interface.
> > > > > > > > > > > > > > > > This method can convert a Catalog instance
> to a
> > > > > > > Map<String,
> > > > > > > > > > > String>
> > > > > > > > > > > > > > > > properties, and the default implementation
> will
> > > > throw
> > > > > > an
> > > > > > > > > > > exception.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > > * Returns a map containing the properties
> of
> > > the
> > > > > > > catalog
> > > > > > > > > > > object.
> > > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > > * @return a map containing the properties
> of
> > > the
> > > > > > > catalog
> > > > > > > > > > > object
> > > > > > > > > > > > > > > > * @throws UnsupportedOperationException if
> > the
> > > > > > > > > implementing
> > > > > > > > > > > > class
> > > > > > > > > > > > > > does
> > > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > > * the default implementation of this
> method
> > > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > > default Map<String, String> toProperties()
> {
> > > > > > > > > > > > > > > > throw new
> > > UnsupportedOperationException("Please
> > > > > > > > implement
> > > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. If the user has configured a
> CatalogStore,
> > > the
> > > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > will be called when registering a Catalog
> > > instance
> > > > > with
> > > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > > catalog).
> > > > > > The
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > will be converted to a Map<String, String>
> > > > properties
> > > > > > and
> > > > > > > > > saved
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > CatalogStore. If some Catalog instances do
> not
> > > > > > implement
> > > > > > > > this
> > > > > > > > > > > > method,
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. If the user does not use a CatalogStore,
> > the
> > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > will not be called, ensuring consistency with
> > the
> > > > > > > original
> > > > > > > > > > > process.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> > catalogs
> > > > and
> > > > > > the
> > > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > > we can start from the Catalog itself and
> > provide
> > > a
> > > > > > > > dedicated
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > > > Catalog
> > > > > {
> > > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > > properties)
> > > > > {
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have no questions about the functionality
> > of
> > > > > > > > > > `CatalogStore`,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > different operations of multiple
> > > > `registerCatalog`
> > > > > or
> > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > `CatalogManager`. The implementation in
> Trino
> > > is
> > > > > also
> > > > > > > the
> > > > > > > > > > same,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > > `createCatalog`,
> > > > > > > > > which
> > > > > > > > > > > > will
> > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > > `registerCatalog`
> > > > > > > or
> > > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > > catalogs`
> > > > > or
> > > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > > catalogStore`. As you mentioned before,
> the
> > > > > > > `Map<String,
> > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`.
> How
> > > > about
> > > > > > > > saving
> > > > > > > > > > the
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > > `CatalogStore
> > > > > > > > > > catalogStore`
> > > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > > when it is registered or added in
> > > > `CatalogManager`?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry
> > for
> > > > the
> > > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > > by my deviation from the original
> > discussion.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> for
> > > > this
> > > > > > FLIP
> > > > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> > > before.
> > > > > > > > However,
> > > > > > > > > > in
> > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > > > CatalogManager
> > > > > > > > > > > itself
> > > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > > need to change much. *We cannot only
> > persist
> > > > > > Catalog
> > > > > > > > > > > instances
> > > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > > > catalogName,
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > not be enough to save Catalogs, because
> in
> > > the
> > > > > end
> > > > > > we
> > > > > > > > > still
> > > > > > > > > > > > need
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > > the configurations corresponding to the
> > > Catalog
> > > > > > > > > instances.
> > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > > interface
> > > > > for
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > persistence. Regarding this part, I also
> > took
> > > > > > > > inspiration
> > > > > > > > > > > from
> > > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> with
> > > the
> > > > > > same
> > > > > > > > name
> > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> > provide
> > > a
> > > > > > > detailed
> > > > > > > > > > > > > description
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > code examples for this part, and add it
> to
> > > the
> > > > > > FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > still
> > > > > looks
> > > > > > > > > confused
> > > > > > > > > > > to
> > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > CatalogStore,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > opinion,
> > > > > you
> > > > > > > can
> > > > > > > > > > decide
> > > > > > > > > > > > to
> > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me.
> I
> > > > don't
> > > > > > see
> > > > > > > > any
> > > > > > > > > > > > problem
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> > > method
> > > > > will
> > > > > > > be
> > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > wondering
> > > > > > > > whether
> > > > > > > > > > user
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > For CREATE CATALOG, my original design
> was
> > to
> > > > add
> > > > > > it
> > > > > > > > > > directly
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > > > existing
> > > > > > > > logic.
> > > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > we can do both: save the configuration to
> > the
> > > > > > > > > CatalogStore
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > > > default
> > > > > > > > > > > implmentation
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > I think it is necessary, otherwise we
> would
> > > > need
> > > > > to
> > > > > > > > > > introduce
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > > > loading.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > > catalog.
> > > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > > catalogName,
> > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > made into an asynchronous interface
> because
> > > > > Catalog
> > > > > > > is
> > > > > > > > > > > already
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> > for
> > > > > > catalog
> > > > > > > > than
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > > > should
> > > > > be
> > > > > > > > lazy
> > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > instead of asynchronous registration. I
> > have
> > > > > > already
> > > > > > > > > > updated
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon
> FY <
> > > > > > > > > > zjureel@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > > > method
> > > > > in
> > > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > confuse users whether a method named
> > > > > `addCatalog`
> > > > > > > or
> > > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > And as you mentioned, the memory
> catalog
> > > is a
> > > > > > > > `cache`,
> > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> > users.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1]
> for
> > > > this
> > > > > > > FLIP.
> > > > > > > > > > Please
> > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > > > discussion
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > > > implement
> > > > > it
> > > > > > > for
> > > > > > > > > > > > different
> > > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think there is a gap between the
> > current
> > > > FLIP
> > > > > > > > design
> > > > > > > > > > and
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > > What about the proposal of the
> discussion
> > > in
> > > > > > thread
> > > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> > with
> > > > the
> > > > > > > same
> > > > > > > > > name
> > > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > complex to me. I think it'll be
> better
> > to
> > > > > > explain
> > > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > java
> > > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > these methods and give a brief
> example
> > in
> > > > > this
> > > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > > still
> > > > > > looks
> > > > > > > > > > confused
> > > > > > > > > > > > to
> > > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > > CatalogStore,
> > > > > > > > > how
> > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > > opinion,
> > > > > > > you
> > > > > > > > > can
> > > > > > > > > > > > decide
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement,
> which
> > > > method
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`?
> I'm
> > > > > > wondering
> > > > > > > > > > whether
> > > > > > > > > > > > user
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to
> provide a
> > > > > default
> > > > > > > > > > > > implmentation
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 4: About asynchronous registration
> for
> > > > > catalog.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > > > CATALOG,
> > > > > > > the
> > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration method is used by
> default.
> > > > > > > > > > > > > > > > > > > > If asynchronous registration is the
> > > default
> > > > > > > > behavior,
> > > > > > > > > > it
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > > > registration
> > > > > > > > just
> > > > > > > > > > > like
> > > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > > `registerCatalog`
> > > > > > > > > be
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration,
> it
> > > may
> > > > > well
> > > > > > > > that
> > > > > > > > > > > CREATE
> > > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > > executes successfully, but then the
> > > > following
> > > > > > > > CREATE
> > > > > > > > > > > TABLE
> > > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > > > I think it's a break change which
> > should
> > > be
> > > > > > > > > highlighted
> > > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > > registration, I
> > > > > > would
> > > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > > will be an executor to open or
> register
> > > > > catalog
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> > initialized
> > > > if
> > > > > it
> > > > > > > has
> > > > > > > > > not
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > > yet. If the initialization process
> > fails,
> > > > > these
> > > > > > > > > > > statements
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > > It looks more like lazy
> initialization
> > > for
> > > > > > > catalog
> > > > > > > > > than
> > > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <
> jinfeng1995@gmail.com
> > >
> > > > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > > > persistence
> > > > > > > of
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > > > interest
> > > > > in
> > > > > > > > this
> > > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > > How to handle a catalog with the
> same
> > > > name
> > > > > > that
> > > > > > > > > > exists
> > > > > > > > > > > > for
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I believe this is a crucial point.
> > Based
> > > on
> > > > > my
> > > > > > > > > current
> > > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > > understanding, the Map<String,
> Catalog>
> > > > > > catalogs
> > > > > > > > will
> > > > > > > > > > > serve
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > > for instantiated catalogs and have
> the
> > > > > highest
> > > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There are three methods that can
> affect
> > > the
> > > > > > > > > Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. registerCatalog(String
> catalogName,
> > > > > Catalog
> > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method puts the catalog instance
> > > into
> > > > > the
> > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> > catalogName)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method removes the catalog
> > instance
> > > > > > > > > corresponding
> > > > > > > > > > to
> > > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > > from the Map<String, Catalog>
> catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > > corresponding
> > > > > > > > catalog
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If
> > the
> > > > > > catalog
> > > > > > > > > does
> > > > > > > > > > > not
> > > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > retrieves the corresponding
> > configuration
> > > > > from
> > > > > > > the
> > > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > > initializes it, and puts the
> > initialized
> > > > > > Catalog
> > > > > > > > > > instance
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The following two methods only modify
> > the
> > > > > > > > > configuration
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > > Map<String,
> > > > > > > > String>
> > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This method saves the properties to
> the
> > > > > > > > catalogStore
> > > > > > > > > > and
> > > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > > whether there is a catalogName with
> the
> > > > same
> > > > > > > name.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > > configuration
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The following are possible conflict
> > > > > scenarios:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > > > already
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore but not in the
> Map<String,
> > > > > > Catalog>,
> > > > > > > > the
> > > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > method
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > > > Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > > > already
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > > both
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > > Catalog>,
> > > > > the
> > > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method
> > will
> > > > > fail.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > > > already
> > > > > > > > exists
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > > addCatalog(String
> > > > > > > > > > catalogName,
> > > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > String> properties) method can
> directly
> > > > save
> > > > > > the
> > > > > > > > > > > properties
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalogStore, but the
> getCatalog(String
> > > > > > > > catalogName)
> > > > > > > > > > > method
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > use the new properties for
> > initialization
> > > > > > because
> > > > > > > > the
> > > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > > catalog instance already exists in
> > > catalogs
> > > > > and
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > > Therefore, using the
> > > > unregisterCatalog(String
> > > > > > > > > > > catalogName)
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > remove the instance corresponding to
> > the
> > > > > > original
> > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > > > `Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > > > Map<String,
> > > > > > > > > > String>
> > > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This could potentially lead to
> > > confusion. I
> > > > > > > suggest
> > > > > > > > > > > > changing
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > method name, perhaps to
> > addCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > > String> properties), as previously
> > > > mentioned
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > > catalogName,Catalog
> > > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Since a catalog is already an
> instance,
> > > > > adding
> > > > > > > the
> > > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > > parameter to the
> registerCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > > method may not necessarily result in
> > lazy
> > > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Do we need to think about
> encryption
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > > > perhaps
> > > > > > the
> > > > > > > > > > > encryption
> > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > > should be implemented in the
> > CatalogStore
> > > > > when
> > > > > > > the
> > > > > > > > > data
> > > > > > > > > > > is
> > > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > > saved. For instance, the
> > FileCatalogStore
> > > > > could
> > > > > > > > > encrypt
> > > > > > > > > > > the
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > > MemoryCatalogStore
> > > > > > > > > > does
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > > MemoryCatalogStore
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > > > configurations
> > > > > > > > in
> > > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > > > asynchronous
> > > > > > > > > loading
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > would need to introduce additional
> > cache.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > What if asynchronous registration
> > > failed?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This is also a critical concern.
> When
> > > > > > executing
> > > > > > > > DDL,
> > > > > > > > > > > DQL,
> > > > > > > > > > > > or
> > > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > > statements that reference a specific
> > > > Catalog,
> > > > > > the
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > > will be initialized if it has not
> been
> > > > > > > initialized
> > > > > > > > > yet.
> > > > > > > > > > > If
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > initialization process fails, these
> > > > > statements
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > be
> > > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog>
> catalogs
> > > and
> > > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > > > catalog
> > > > > > > name,
> > > > > > > > > but
> > > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > This issue can be resolved by
> referring
> > > to
> > > > > the
> > > > > > > > > previous
> > > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > key principle is that the Catalog
> that
> > > has
> > > > > been
> > > > > > > > most
> > > > > > > > > > > > recently
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > > will be given priority for subsequent
> > > use.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM
> Xianxun
> > > Ye
> > > > <
> > > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > > > believe
> > > > > > this
> > > > > > > > > Flip
> > > > > > > > > > > > could
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > > > manages a
> > > > > > lot
> > > > > > > > of
> > > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> > because
> > > of
> > > > > > > > loading a
> > > > > > > > > > > > number
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> > > Flink
> > > > > SQL
> > > > > > > > > through
> > > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> > improve
> > > > the
> > > > > > > speed
> > > > > > > > of
> > > > > > > > > > > task
> > > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > > > questions:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> > CREATE
> > > > > > CATALOG,
> > > > > > > > the
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > > registration method is used by
> default.
> > > > What
> > > > > if
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > > failed? And how to implement
> > asynchronous
> > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. I also have the same question
> with
> > > > > > Shammon’s
> > > > > > > > > first
> > > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > > CatalogStore
> > > > > > > > > > > catalogstore
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> > catalog
> > > > > name,
> > > > > > > but
> > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > different catalog instances. For
> > example:
> > > > > > > > catalog1->
> > > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog
> > in
> > > > the
> > > > > > > > > > > catalogstore.
> > > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> > `catalogs`
> > > > and
> > > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration
Posted by Jark Wu <im...@gmail.com>.
Hi Feng,
This is a useful FLIP. Thanks for starting this discussion.
The current design looks pretty good to me. I just have some minor comments.
1. How to register the CatalogStore for Table API? I think the CatalogStore
should be immutable once TableEnv is created. Otherwise, there might be
some data inconsistencies when CatalogStore is changed.
2. Why does the CatalogStoreFactory interface only have a default method,
not an interface method?
3. Please mention the alternative API in Javadoc for the deprecated
`registerCatalog`.
4. In the "Compatibility" section, would be better to mention the changed
behavior of CREATE CATALOG statement if FileCatalogStore (or other
persisted catalog store) is used.
Best,
Jark
On Thu, 1 Jun 2023 at 11:26, Feng Jin <ji...@gmail.com> wrote:
> Hi ， thanks all for reviewing the flip.
>
> @Ron
>
> > Regarding the CatalogStoreFactory#createCatalogStore method, do we need
> to provide a default implementation?
>
> Yes, we will provide a default InMemoryCatalogStoreFactory to create an
> InMemoryCatalogStore.
>
> > If we get a Catalog from CatalogStore, after initializing it, whether we
> put it in Map<String, Catalog> catalogs again?
>
> Yes, in the current design, catalogs are stored as snapshots, and once
> initialized, the Catalog will be placed in the Map<String, Catalog>
> catalogs.
> Subsequently, the Map<String, Catalog> catalogs will be the primary source
> for obtaining the corresponding Catalog.
>
> > how about renaming them to `catalog.store.type` and
> `catalog.store.path`?
>
> I think it is okay. Adding "sql" at the beginning may seem a bit strange. I
> will update the FLIP.
>
>
>
> @Shammon
>
> Thank you for the review. I have made the necessary corrections.
> Regarding the modifications made to the Public Interface, I have also
> included the relevant changes to the `TableEnvironment`.
>
>
> Best,
> Feng
>
>
> On Wed, May 31, 2023 at 5:02 PM Shammon FY <zj...@gmail.com> wrote:
>
> > Hi feng,
> >
> > Thanks for updating, I have some minor comments
> >
> > 1. The modification of `CatalogManager` should not be in `Public
> > Interfaces`, it is not a public interface.
> >
> > 2. `@PublicEvolving` should be added for `CatalogStore` and
> > `CatalogStoreFactory`
> >
> > 3. The code `Optional<Catalog> optionalDescriptor =
> > catalogStore.get(catalogName);` in the `CatalogManager` should be
> > `Optional<CatalogDescriptor> optionalDescriptor =
> > catalogStore.get(catalogName);`
> >
> > Best,
> > Shammon FY
> >
> >
> > On Wed, May 31, 2023 at 2:24 PM liu ron <ro...@gmail.com> wrote:
> >
> > > Hi, Feng
> > >
> > > Thanks for driving this FLIP, this proposal is very useful for catalog
> > > management.
> > > I have some small questions:
> > >
> > > 1. Regarding the CatalogStoreFactory#createCatalogStore method, do we
> > need
> > > to provide a default implementation?
> > > 2. If we get Catalog from CatalogStore, after initializing it, whether
> we
> > > put it to Map<String, Catalog> catalogs again?
> > > 3. Regarding the options `sql.catalog.store.type` and
> > > `sql.catalog.store.file.path`, how about renaming them to
> > > `catalog.store.type` and `catalog.store.path`?
> > >
> > > Best,
> > > Ron
> > >
> > > Feng Jin <ji...@gmail.com> 于2023年5月29日周一 21:19写道：
> > >
> > > > Hi yuxia
> > > >
> > > > > But from the code in Proposed Changes, once we register the
> Catalog,
> > > we
> > > > initialize it and open it. right?
> > > >
> > > > Yes, In order to avoid inconsistent semantics of the original CREATE
> > > > CATALOG DDL, Catalog will be directly initialized in registerCatalog
> so
> > > > that parameter validation can be performed.
> > > >
> > > > In the current design, lazy initialization is mainly reflected in
> > > > getCatalog. If CatalogStore has already saved some catalog
> > > configurations,
> > > > only initialization is required in getCatalog.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Mon, May 29, 2023 at 8:27 PM yuxia <lu...@alumni.sjtu.edu.cn>
> > > wrote:
> > > >
> > > > > Hi, Feng.
> > > > > I'm trying to understanding the meaning of *lazy initialization*.
> If
> > > i'm
> > > > > wrong, please correct me.
> > > > >
> > > > > IIUC, lazy initialization means only you need to access the
> catalog,
> > > then
> > > > > you initialize it. But from the code in Proposed Changes, once we
> > > > register
> > > > > the Catalog,
> > > > > we initialize it and open it. right?
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > ----- 原始邮件 -----
> > > > > 发件人: "Jing Ge" <ji...@ververica.com.INVALID>
> > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > 发送时间: 星期一, 2023年 5 月 29日 下午 5:12:46
> > > > > 主题: Re: [DISCUSS] FLIP 295: Support persistence of Catalog
> > > configuration
> > > > > and asynchronous registration
> > > > >
> > > > > Hi Feng,
> > > > >
> > > > > Thanks for your effort! +1 for the proposal.
> > > > >
> > > > > One of the major changes is that current design will provide
> > > > > Map<String,Catalog> catalogs as a snapshot instead of a cache,
> which
> > > > means
> > > > > once it has been initialized, any changes done by other sessions
> will
> > > not
> > > > > affect it. Point 6 described follow-up options for further
> > improvement.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Mon, May 29, 2023 at 5:31 AM Feng Jin <ji...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi all, I would like to update you on the latest progress of the
> > > FLIP.
> > > > > >
> > > > > >
> > > > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai
> Fang
> > > > and I
> > > > > > had an offline discussion regarding the overall solution for
> Flink
> > > > > > CatalogStore. We have reached a consensus and I have updated the
> > > final
> > > > > > solution in FLIP.
> > > > > >
> > > > > > Next, let me briefly describe the entire design:
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Introduce CatalogDescriptor to store catalog configuration
> > similar
> > > > to
> > > > > > TableDescriptor.
> > > > > > 2.
> > > > > >
> > > > > > The two key functions of CatalogStore - void
> storeCatalog(String
> > > > > > catalogName, CatalogDescriptor) and CatalogDescriptor
> > > > > getCatalog(String)
> > > > > > will both use CatalogDescriptor instead of Catalog instance.
> > This
> > > > way,
> > > > > > CatalogStore will only be responsible for saving and
> retrieving
> > > > > catalog
> > > > > > configurations without having to initialize catalogs.
> > > > > > 3.
> > > > > >
> > > > > > The default registerCatalog(String catalogName, Catalog
> catalog)
> > > > > > function in CatalogManager will be marked as deprecated.
> > > > > > 4.
> > > > > >
> > > > > > A new function registerCatalog(String catalogName,
> > > CatalogDescriptor
> > > > > > catalog) will be added to serve as the default registration
> > > function
> > > > > for
> > > > > > catalogs in CatalogManager.
> > > > > > 5.
> > > > > >
> > > > > > Map<String,Catalog> catalogs in CataloManager will remain
> > > unchanged
> > > > > and
> > > > > > save initialized catalogs.This means that deletion operations
> > from
> > > > one
> > > > > > session won't synchronize with other sessions.
> > > > > > 6.
> > > > > >
> > > > > > To support multi-session synchronization scenarios for
> deletions
> > > > later
> > > > > > on we should make Map<String,Catalog>catalogs
> configurable.There
> > > may
> > > > > be
> > > > > > three possible situations:
> > > > > >
> > > > > > a.Default caching of all initialized catalogs
> > > > > >
> > > > > > b.Introduction of LRU cache logic which can automatically
> clear
> > > > > > long-unused catalogs.
> > > > > >
> > > > > > c.No caching of any instances; each call to getCatalog
> creates a
> > > new
> > > > > > instance.
> > > > > >
> > > > > >
> > > > > > This is the document for discussion:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1HRJNd4_id7i6cUxGnAybmYZIwl5g1SmZCOzGdUz-6lU/edit
> > > > > >
> > > > > > This is the final proposal document:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations
> > > > > >
> > > > > >
> > > > > > Thank you very much for your attention and suggestions on this
> > > FLIP. A
> > > > > > special thanks to Hang Ruan for his suggestions on the entire
> > design
> > > > and
> > > > > > organizing offline discussions.
> > > > > >
> > > > > > If you have any further suggestions or feedback about this FLIP
> > > please
> > > > > feel
> > > > > > free to share.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Feng
> > > > > >
> > > > > > On Sat, May 6, 2023 at 8:32 PM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >
> > > > > > > Thanks for improving the FLIP. It looks good to me. We could
> > still
> > > > > > > reconsider in the future how to provide more common built-in
> > cache
> > > > > > > functionality in CatalogManager so that not every CatalogSotre
> > > > > > > implementation has to take care of it.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Thu, May 4, 2023 at 1:47 PM Feng Jin <jinfeng1995@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Jing,
> > > > > > > >
> > > > > > > > Thanks for your reply.
> > > > > > > >
> > > > > > > > > There might be more such issues. I would suggest you
> > > completely
> > > > > walk
> > > > > > > > through the FLIP again and fix those issues
> > > > > > > >
> > > > > > > > I am very sorry for my carelessness and at the same time, I
> > > greatly
> > > > > > > > appreciate your careful review. I have thoroughly checked the
> > > > entire
> > > > > > FLIP
> > > > > > > > and made corrections to these issues.
> > > > > > > >
> > > > > > > >
> > > > > > > > > If I am not mistaken, with the current FLIP design,
> > > > > CatalogManager
> > > > > > > > could work without Optional CatalogStore being configured.
> > > > > > > >
> > > > > > > > Yes, in the original design, CatalogStore was not necessary
> > > because
> > > > > > > > CatalogManager used Map<String, Catalog> catalogs to store
> > > catalog
> > > > > > > > instances.
> > > > > > > > However, this caused inconsistency issues. Therefore, I
> > modified
> > > > this
> > > > > > > part
> > > > > > > > of the design and removed Map<String, Catalog> catalogs from
> > > > > > > > CatalogManager.
> > > > > > > > At the same time, InMemoryCatalog will serve as the default
> > > > > > CatalogStore
> > > > > > > > to save catalogs in memory and replace the functionality of
> > > > > > > > Map<String,Catalog>catalogs.
> > > > > > > > The previous plan that kept Map<String,Catalog>catalogs has
> > been
> > > > > moved
> > > > > > to
> > > > > > > > Rejected Alternatives.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng
> > > > > > > >
> > > > > > > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Feng,
> > > > > > > > >
> > > > > > > > > There are still many places contain inconsistent content,
> > e.g.
> > > > > > > > >
> > > > > > > > > 1. "Asynchronous registration" is still used.
> > > > > > > > > 2. The java comment of the method registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog catalog) in CatalogManager does not tell what the
> > > method
> > > > is
> > > > > > > > doing.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > There might be more such issues. I would suggest you
> > completely
> > > > > walk
> > > > > > > > > through the FLIP again and fix those issues.
> > > > > > > > >
> > > > > > > > > About the inMemoryCatalogStore, do you mean that you will
> > build
> > > > the
> > > > > > > cache
> > > > > > > > > functionality in the CatalogStore? This is a very different
> > > > design
> > > > > > > > concept
> > > > > > > > > from what the current FLIP described. If I am not mistaken,
> > > with
> > > > > the
> > > > > > > > > current FLIP design, CatalogManager could work without
> > Optional
> > > > > > > > > CatalogStore being configured. That is the reason why I
> > > mentioned
> > > > > in
> > > > > > > the
> > > > > > > > > last email that the example code wrt the Optional
> > CatalogStore
> > > is
> > > > > not
> > > > > > > > > correct.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Thu, Apr 27, 2023 at 3:55 PM Feng Jin <
> > > jinfeng1995@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jing
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > There are still some NIT issues in the FLIP
> > > > > > > > > >
> > > > > > > > > > Thank you very much for the careful review. I have
> already
> > > made
> > > > > the
> > > > > > > > > > relevant changes.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > > scenarios, I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > sure if this is the intended behaviour
> > > > > > > > > >
> > > > > > > > > > Currently, there are conflicts in multiple scenarios with
> > the
> > > > > > current
> > > > > > > > > > design. I am thinking whether we should remove
> 'Map<String,
> > > > > > Catalog>'
> > > > > > > > and
> > > > > > > > > > make Cache the default behavior of InMemoryCatalogStore.
> > This
> > > > > way,
> > > > > > > > users
> > > > > > > > > > can implement their own CatalogStore to achieve
> > > multi-instance
> > > > > > > > > > non-conflicting scenarios. What do you think?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Feng,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for working on the FLIP. There are still some
> NIT
> > > > issues
> > > > > > in
> > > > > > > > the
> > > > > > > > > > FLIP
> > > > > > > > > > > like:
> > > > > > > > > > >
> > > > > > > > > > > 1. Optional<CatalogStore> catalogStore has been used as
> > > > > > > CatalogStore
> > > > > > > > > > > instead of Optional in the code example. It should be
> > fine
> > > to
> > > > > use
> > > > > > > it
> > > > > > > > as
> > > > > > > > > > > pseudo code for now and update it after you submit the
> > PR.
> > > > > > > > > > > 2. addCatalog(...) is still used somewhere in the
> > rejected
> > > > > > section
> > > > > > > > > which
> > > > > > > > > > > should be persistContext(...) to keep it consistent.
> > > > > > > > > > >
> > > > > > > > > > > Speaking of the conflict issues in the multi-instance
> > > > > scenarios,
> > > > > > I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > > > sure if this is the intended behaviour. If Map<String,
> > > > Catalog>
> > > > > > > > > catalogs
> > > > > > > > > > is
> > > > > > > > > > > used as a cache, it should be invalid, once the related
> > > > catalog
> > > > > > has
> > > > > > > > > been
> > > > > > > > > > > removed from the CatalogStore by another instance. Did
> I
> > > miss
> > > > > > > > > something?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Apr 13, 2023 at 4:40 PM Feng Jin <
> > > > > jinfeng1995@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing,Shammon
> > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > >
> > > > > > > > > > > > @Jing
> > > > > > > > > > > >
> > > > > > > > > > > > > How about persistCatalog()?
> > > > > > > > > > > > I think this is a good function name, I have updated
> it
> > > in
> > > > > the
> > > > > > > > > > > > documentation.
> > > > > > > > > > > >
> > > > > > > > > > > > > Some common cache features should be implemented
> > > > > > > > > > > > Thank you for the suggestion. If alternative 1 turns
> > out
> > > to
> > > > > be
> > > > > > > more
> > > > > > > > > > > > appropriate later, I will improve this part of the
> > > content.
> > > > > > > > > > > >
> > > > > > > > > > > > > As the above discussion moves forward, the option 2
> > > > > solution
> > > > > > > > looks
> > > > > > > > > > more
> > > > > > > > > > > > like a replacement of option 1
> > > > > > > > > > > > Yes, after discussing with Shammon offline, we think
> > that
> > > > > > > solution
> > > > > > > > 2
> > > > > > > > > > > might
> > > > > > > > > > > > be more suitable and also avoid any inconsistency
> > issues.
> > > > > > > > > > > >
> > > > > > > > > > > > > There are some inconsistent descriptions in the
> > > content.
> > > > > > Would
> > > > > > > > you
> > > > > > > > > > > like
> > > > > > > > > > > > to clean them up?
> > > > > > > > > > > > I will do my best to improve the document and
> > appreciate
> > > > your
> > > > > > > > > > > suggestions.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > can you put the unselected option in `Rejected
> > > > > Alternatives`
> > > > > > > > > > > > Sure, I have moved it to the `Rejected Alternatives`.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best
> > > > > > > > > > > > Feng
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY <
> > > > > zjureel@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your update.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I found there are two options in `Proposed
> Changes`,
> > > can
> > > > > you
> > > > > > > put
> > > > > > > > > the
> > > > > > > > > > > > > unselected option in `Rejected Alternatives`? I
> think
> > > > this
> > > > > > may
> > > > > > > > help
> > > > > > > > > > us
> > > > > > > > > > > > > better understand your proposal
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Apr 13, 2023 at 4:49 AM Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for raising this FLIP. I am still confused
> > > after
> > > > > > > > > completely
> > > > > > > > > > > > > reading
> > > > > > > > > > > > > > the thread with following questions:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Naming confusion - registerCatalog() and
> > > > addCatalog()
> > > > > > have
> > > > > > > > no
> > > > > > > > > > big
> > > > > > > > > > > > > > difference based on their names. One of them is
> > > > > responsible
> > > > > > > for
> > > > > > > > > > data
> > > > > > > > > > > > > > persistence. How about persistCatalog()?
> > > > > > > > > > > > > > 2. As you mentioned that Map<String, Catalog>
> > > catalogs
> > > > is
> > > > > > > used
> > > > > > > > > as a
> > > > > > > > > > > > cache
> > > > > > > > > > > > > > and catalogStore is used for data persistence. I
> > > would
> > > > > > > suggest
> > > > > > > > > > > > describing
> > > > > > > > > > > > > > their purpose conceptually and clearly in the
> FLIP.
> > > > Some
> > > > > > > common
> > > > > > > > > > cache
> > > > > > > > > > > > > > features should be implemented, i.e. data in the
> > > cache
> > > > > and
> > > > > > > the
> > > > > > > > > > store
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > be consistent. Same Catalog instance should be
> > found
> > > in
> > > > > the
> > > > > > > > store
> > > > > > > > > > and
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the cache(either it has been initialized or it
> will
> > > be
> > > > > lazy
> > > > > > > > > > > > initialized)
> > > > > > > > > > > > > > for the same catalog name. The consistency will
> be
> > > > taken
> > > > > > care
> > > > > > > > of
> > > > > > > > > > > while
> > > > > > > > > > > > > > updating the catalog.
> > > > > > > > > > > > > > 3. As the above discussion moves forward, the
> > option
> > > 2
> > > > > > > solution
> > > > > > > > > > looks
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > like a replacement of option 1, because, afaiu,
> > > issues
> > > > > > > > mentioned
> > > > > > > > > > > > > > previously with option 1 are not solved yet. Do
> you
> > > > still
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > > > propose
> > > > > > > > > > > > > > both options and ask for suggestions for both of
> > > them?
> > > > > > > > > > > > > > 4. After you updated the FLIP, there are some
> > > > > inconsistent
> > > > > > > > > > > descriptions
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the content. Would you like to clean them up?
> > > Thanks!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Jing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Apr 7, 2023 at 9:24 AM Feng Jin <
> > > > > > > jinfeng1995@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > hi Shammon
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you for your response, and I completely
> > agree
> > > > > with
> > > > > > > your
> > > > > > > > > > point
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > view.
> > > > > > > > > > > > > > > Initially, I may have over complicated the
> whole
> > > > issue.
> > > > > > > First
> > > > > > > > > and
> > > > > > > > > > > > > > foremost,
> > > > > > > > > > > > > > > we need to consider the persistence of the
> > > Catalog's
> > > > > > > > > > Configuration.
> > > > > > > > > > > > > > > If we only need to provide persistence for
> > Catalog
> > > > > > > > > Configuration,
> > > > > > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > add a toConfiguration method to the Catalog
> > > > interface.
> > > > > > > > > > > > > > > This method can convert a Catalog instance to a
> > > > > > Map<String,
> > > > > > > > > > String>
> > > > > > > > > > > > > > > properties, and the default implementation will
> > > throw
> > > > > an
> > > > > > > > > > exception.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > public interface Catalog {
> > > > > > > > > > > > > > > /**
> > > > > > > > > > > > > > > * Returns a map containing the properties of
> > the
> > > > > > catalog
> > > > > > > > > > object.
> > > > > > > > > > > > > > > *
> > > > > > > > > > > > > > > * @return a map containing the properties of
> > the
> > > > > > catalog
> > > > > > > > > > object
> > > > > > > > > > > > > > > * @throws UnsupportedOperationException if
> the
> > > > > > > > implementing
> > > > > > > > > > > class
> > > > > > > > > > > > > does
> > > > > > > > > > > > > > > not override
> > > > > > > > > > > > > > > * the default implementation of this method
> > > > > > > > > > > > > > > */
> > > > > > > > > > > > > > > default Map<String, String> toProperties() {
> > > > > > > > > > > > > > > throw new
> > UnsupportedOperationException("Please
> > > > > > > implement
> > > > > > > > > > > > > > toProperties
> > > > > > > > > > > > > > > for this catalog");
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The specific process is as follows:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. If the user has configured a CatalogStore,
> > the
> > > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > will be called when registering a Catalog
> > instance
> > > > with
> > > > > > > > > > > > > > > registerCatalog(String catalogName, Catalog
> > > catalog).
> > > > > The
> > > > > > > > > Catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > will be converted to a Map<String, String>
> > > properties
> > > > > and
> > > > > > > > saved
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > CatalogStore. If some Catalog instances do not
> > > > > implement
> > > > > > > this
> > > > > > > > > > > method,
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > exception will be thrown.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. If the user does not use a CatalogStore,
> the
> > > > > > > > > > toConfiguration()
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > will not be called, ensuring consistency with
> the
> > > > > > original
> > > > > > > > > > process.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. Saving both the Map<String, Catalog>
> catalogs
> > > and
> > > > > the
> > > > > > > > > > > CatalogStore
> > > > > > > > > > > > > at
> > > > > > > > > > > > > > > the same time can also avoid conflicts
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For lazy initialization:
> > > > > > > > > > > > > > > we can start from the Catalog itself and
> provide
> > a
> > > > > > > dedicated
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > implementation for lazy loading, such as
> > > > > > > ConfigurationCatalog
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > public class ConfigurationCatalog implements
> > > Catalog
> > > > {
> > > > > > > > > > > > > > > ConfigurationCatalog(Map<String, String>
> > > properties)
> > > > {
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have added this design to the FLIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Feng
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Apr 6, 2023 at 10:03 AM Shammon FY <
> > > > > > > > zjureel@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your answer.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have no questions about the functionality
> of
> > > > > > > > > `CatalogStore`,
> > > > > > > > > > > but
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > different operations of multiple
> > > `registerCatalog`
> > > > or
> > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > `CatalogManager`. The implementation in Trino
> > is
> > > > also
> > > > > > the
> > > > > > > > > same,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > `CatalogManager` in trino only has one
> > > > > `createCatalog`,
> > > > > > > > which
> > > > > > > > > > > will
> > > > > > > > > > > > > save
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > catalog to memory and `CatalogStore`.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think we don't need to add another
> > > > > `registerCatalog`
> > > > > > or
> > > > > > > > > > > > > `addCatalog`
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > save a catalog in `Map<String, Catalog>
> > catalogs`
> > > > or
> > > > > > > > > > > `CatalogStore
> > > > > > > > > > > > > > > > catalogStore`. As you mentioned before, the
> > > > > > `Map<String,
> > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > catalogs` is a cache in `CatalogManager`. How
> > > about
> > > > > > > saving
> > > > > > > > > the
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > `Map<String, Catalog> catalogs` and
> > `CatalogStore
> > > > > > > > > catalogStore`
> > > > > > > > > > > > > > together
> > > > > > > > > > > > > > > > when it is registered or added in
> > > `CatalogManager`?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 5:22 PM Feng Jin <
> > > > > > > > > jinfeng1995@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you for your reply. I am very sorry
> for
> > > the
> > > > > > > > > > > > misunderstanding
> > > > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > > by my deviation from the original
> discussion.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > > this
> > > > > FLIP
> > > > > > > > > > > > > > > > > Yes, there was indeed such a discussion
> > before.
> > > > > > > However,
> > > > > > > > > in
> > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > whole solution, I found that the logic of
> > > > > > > CatalogManager
> > > > > > > > > > itself
> > > > > > > > > > > > > > doesn't
> > > > > > > > > > > > > > > > > need to change much. *We cannot only
> persist
> > > > > Catalog
> > > > > > > > > > instances
> > > > > > > > > > > > > > > > themselves*,
> > > > > > > > > > > > > > > > > so exposing only registerCatalog(String
> > > > > catalogName,
> > > > > > > > > Catalog
> > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > not be enough to save Catalogs, because in
> > the
> > > > end
> > > > > we
> > > > > > > > still
> > > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > save
> > > > > > > > > > > > > > > > > the configurations corresponding to the
> > Catalog
> > > > > > > > instances.
> > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > decided to introduce the CatalogStore
> > interface
> > > > for
> > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > persistence. Regarding this part, I also
> took
> > > > > > > inspiration
> > > > > > > > > > from
> > > > > > > > > > > > > > Trino's
> > > > > > > > > > > > > > > > > implementation[1].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @yuxia
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog with
> > the
> > > > > same
> > > > > > > name
> > > > > > > > > > > looks a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > complex to me.
> > > > > > > > > > > > > > > > > Thank you for the suggestion. I will
> provide
> > a
> > > > > > detailed
> > > > > > > > > > > > description
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > code examples for this part, and add it to
> > the
> > > > > FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> still
> > > > looks
> > > > > > > > confused
> > > > > > > > > > to
> > > > > > > > > > > > me.
> > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > CatalogStore,
> > > > > > how
> > > > > > > > > about
> > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > opinion,
> > > > you
> > > > > > can
> > > > > > > > > decide
> > > > > > > > > > > to
> > > > > > > > > > > > > take
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > StoreCatalog looks more intuitive to me. I
> > > don't
> > > > > see
> > > > > > > any
> > > > > > > > > > > problem
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3.For CREATE CATALOG statement, which
> > method
> > > > will
> > > > > > be
> > > > > > > > > > called?
> > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > wondering
> > > > > > > whether
> > > > > > > > > user
> > > > > > > > > > > can
> > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > For CREATE CATALOG, my original design was
> to
> > > add
> > > > > it
> > > > > > > > > directly
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > CatalogStore, but this would disrupt the
> > > existing
> > > > > > > logic.
> > > > > > > > > > > > > Therefore, I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > we can do both: save the configuration to
> the
> > > > > > > > CatalogStore
> > > > > > > > > > and
> > > > > > > > > > > > > > > initialize
> > > > > > > > > > > > > > > > > the Catalog instance at the same time
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. Is it really neccessary to provide a
> > > default
> > > > > > > > > > implmentation
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > I think it is necessary, otherwise we would
> > > need
> > > > to
> > > > > > > > > introduce
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > additional
> > > > > > > > > > > > > > > > > Map to store the configuration for lazy
> > > loading.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > catalog.
> > > > > > > > > > > > > > > > > I don't think registerCatalog(String
> > > catalogName,
> > > > > > > Catalog
> > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > made into an asynchronous interface because
> > > > Catalog
> > > > > > is
> > > > > > > > > > already
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > instance.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> for
> > > > > catalog
> > > > > > > than
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > Indeed, my description was inaccurate. It
> > > should
> > > > be
> > > > > > > lazy
> > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > instead of asynchronous registration. I
> have
> > > > > already
> > > > > > > > > updated
> > > > > > > > > > > the
> > > > > > > > > > > > > > title
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/CatalogStore.java
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 4:27 PM Shammon FY <
> > > > > > > > > zjureel@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Feng
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think if there is a `registerCatalog`
> > > method
> > > > in
> > > > > > > > > > > > > `CatalogManager`,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > confuse users whether a method named
> > > > `addCatalog`
> > > > > > or
> > > > > > > > > > > > > `storeCatalog`
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > added.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > And as you mentioned, the memory catalog
> > is a
> > > > > > > `cache`,
> > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > concept
> > > > > > > > > > > > > > > > > > of `cache` should not be exposed to
> users.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I found there is a pre-discussion [1] for
> > > this
> > > > > > FLIP.
> > > > > > > > > Please
> > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > I'm wrong, IIUC, the conclusion of that
> > > > > discussion
> > > > > > is
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > > > > > > > `CatalogManager` as an interface and
> > > implement
> > > > it
> > > > > > for
> > > > > > > > > > > different
> > > > > > > > > > > > > > > stores
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > as memory, file and external system.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think there is a gap between the
> current
> > > FLIP
> > > > > > > design
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > > > > > conclusion.
> > > > > > > > > > > > > > > > > > What about the proposal of the discussion
> > in
> > > > > thread
> > > > > > > > [1] ?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > >
> > https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Shammon FY
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Apr 4, 2023 at 3:41 PM yuxia <
> > > > > > > > > > > > > luoyuxia@alumni.sjtu.edu.cn>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks Feng for driving this FLIP.
> > > > > > > > > > > > > > > > > > > I have few comments:
> > > > > > > > > > > > > > > > > > > 1: The mechanism of handling catalog
> with
> > > the
> > > > > > same
> > > > > > > > name
> > > > > > > > > > > > looks a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > complex to me. I think it'll be better
> to
> > > > > explain
> > > > > > > it
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > java
> > > > > > > > > > > > > > > doc
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > these methods and give a brief example
> in
> > > > this
> > > > > > > FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2: TBH, the method name `addCatalog`
> > still
> > > > > looks
> > > > > > > > > confused
> > > > > > > > > > > to
> > > > > > > > > > > > > me.
> > > > > > > > > > > > > > > > IIUC,
> > > > > > > > > > > > > > > > > > > this method is for storing catalog to
> > > > > > CatalogStore,
> > > > > > > > how
> > > > > > > > > > > about
> > > > > > > > > > > > > > > > renaming
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > to `storeCatalog`? It's very personal
> > > > opinion,
> > > > > > you
> > > > > > > > can
> > > > > > > > > > > decide
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > take
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > not by your self.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3: For CREATE CATALOG statement, which
> > > method
> > > > > > will
> > > > > > > be
> > > > > > > > > > > called?
> > > > > > > > > > > > > > > > > > > `registerCatalog` or `addCatalog`? I'm
> > > > > wondering
> > > > > > > > > whether
> > > > > > > > > > > user
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > add a
> > > > > > > > > > > > > > > > > > > catalog to store with SQL stement.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3: Is it really neccessary to provide a
> > > > default
> > > > > > > > > > > implmentation
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > interface `CatalogStoreFactory`?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 4: About asynchronous registration for
> > > > catalog.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > When creating a catalog with CREATE
> > > > CATALOG,
> > > > > > the
> > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration method is used by default.
> > > > > > > > > > > > > > > > > > > If asynchronous registration is the
> > default
> > > > > > > behavior,
> > > > > > > > > it
> > > > > > > > > > > > there
> > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > way
> > > > > > > > > > > > > > > > > > > that user can switch to synchronous
> > > > > registration
> > > > > > > just
> > > > > > > > > > like
> > > > > > > > > > > > > > before?
> > > > > > > > > > > > > > > > > > > Will both method `addCatalog` and
> > > > > > `registerCatalog`
> > > > > > > > be
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > IIUC, in asynchronous registration, it
> > may
> > > > well
> > > > > > > that
> > > > > > > > > > CREATE
> > > > > > > > > > > > > > CATALOG
> > > > > > > > > > > > > > > > > > > executes successfully, but then the
> > > following
> > > > > > > CREATE
> > > > > > > > > > TABLE
> > > > > > > > > > > > > > > statement
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > fail for the catalog fail to open.
> > > > > > > > > > > > > > > > > > > I think it's a break change which
> should
> > be
> > > > > > > > highlighted
> > > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > FLIP,
> > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > be in compatibility part.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BTW, by saying asynchronous
> > registration, I
> > > > > would
> > > > > > > > like
> > > > > > > > > to
> > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > > will be an executor to open or register
> > > > catalog
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > > > > background,
> > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > your previous comments,
> > > > > > > > > > > > > > > > > > > "the Catalog instance will be
> initialized
> > > if
> > > > it
> > > > > > has
> > > > > > > > not
> > > > > > > > > > > been
> > > > > > > > > > > > > > > > > initialized
> > > > > > > > > > > > > > > > > > > yet. If the initialization process
> fails,
> > > > these
> > > > > > > > > > statements
> > > > > > > > > > > > will
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > executed successfully."
> > > > > > > > > > > > > > > > > > > It looks more like lazy initialization
> > for
> > > > > > catalog
> > > > > > > > than
> > > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration, right?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > Yuxia
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > ----- 原始邮件 -----
> > > > > > > > > > > > > > > > > > > 发件人: "Feng Jin" <jinfeng1995@gmail.com
> >
> > > > > > > > > > > > > > > > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > > > > > > > > > > > > > > > 发送时间: 星期一, 2023年 4 月 03日 下午 3:27:45
> > > > > > > > > > > > > > > > > > > 主题: Re: [DISCUSS] FLIP 295: Support
> > > > persistence
> > > > > > of
> > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > and asynchronous registration
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi everyone, Thank you all for your
> > > interest
> > > > in
> > > > > > > this
> > > > > > > > > > > DISCUSS.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Shammon
> > > > > > > > > > > > > > > > > > > > How to handle a catalog with the same
> > > name
> > > > > that
> > > > > > > > > exists
> > > > > > > > > > > for
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > them?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I believe this is a crucial point.
> Based
> > on
> > > > my
> > > > > > > > current
> > > > > > > > > > > > personal
> > > > > > > > > > > > > > > > > > > understanding, the Map<String, Catalog>
> > > > > catalogs
> > > > > > > will
> > > > > > > > > > serve
> > > > > > > > > > > > as
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > cache
> > > > > > > > > > > > > > > > > > > for instantiated catalogs and have the
> > > > highest
> > > > > > > > > priority.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There are three methods that can affect
> > the
> > > > > > > > Map<String,
> > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > catalogs:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. registerCatalog(String catalogName,
> > > > Catalog
> > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method puts the catalog instance
> > into
> > > > the
> > > > > > > > > > Map<String,
> > > > > > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. unregisterCatalog(String
> catalogName)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method removes the catalog
> instance
> > > > > > > > corresponding
> > > > > > > > > to
> > > > > > > > > > > > > > > catalogName
> > > > > > > > > > > > > > > > > > > from the Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. getCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method first retrieves the
> > > corresponding
> > > > > > > catalog
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > the Map<String, Catalog> catalogs. If
> the
> > > > > catalog
> > > > > > > > does
> > > > > > > > > > not
> > > > > > > > > > > > > exist,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > retrieves the corresponding
> configuration
> > > > from
> > > > > > the
> > > > > > > > > > > > > CatalogStore,
> > > > > > > > > > > > > > > > > > > initializes it, and puts the
> initialized
> > > > > Catalog
> > > > > > > > > instance
> > > > > > > > > > > > into
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The following two methods only modify
> the
> > > > > > > > configuration
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. addCatalog(String catalogName,
> > > Map<String,
> > > > > > > String>
> > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This method saves the properties to the
> > > > > > > catalogStore
> > > > > > > > > and
> > > > > > > > > > > > checks
> > > > > > > > > > > > > > > > > > > whether there is a catalogName with the
> > > same
> > > > > > name.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. removeCatalog(String catalogName)
> > > > > > > > > > > > > > > > > > > This method removes the specified
> > > > configuration
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > specified
> > > > > > > > > > > > > > > > > > > catalogName in the catalogStore.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The following are possible conflict
> > > > scenarios:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. When the corresponding catalogName
> > > already
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore but not in the Map<String,
> > > > > Catalog>,
> > > > > > > the
> > > > > > > > > > > > > > > > > > > registerCatalog(String catalogName,
> > Catalog
> > > > > > > catalog)
> > > > > > > > > > method
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > succeed and be directly saved to the
> > > > > Map<String,
> > > > > > > > > Catalog>
> > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2. When the corresponding catalogName
> > > already
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > > both
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogStore and the Map<String,
> > Catalog>,
> > > > the
> > > > > > > > > > > > > > > registerCatalog(String
> > > > > > > > > > > > > > > > > > > catalogName, Catalog catalog) method
> will
> > > > fail.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 3. When the corresponding catalogName
> > > already
> > > > > > > exists
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Map<String, Catalog>, the
> > addCatalog(String
> > > > > > > > > catalogName,
> > > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > String> properties) method can directly
> > > save
> > > > > the
> > > > > > > > > > properties
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalogStore, but the getCatalog(String
> > > > > > > catalogName)
> > > > > > > > > > method
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > use the new properties for
> initialization
> > > > > because
> > > > > > > the
> > > > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > > > > > > catalog instance already exists in
> > catalogs
> > > > and
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > > > > > prioritized.
> > > > > > > > > > > > > > > > > > > Therefore, using the
> > > unregisterCatalog(String
> > > > > > > > > > catalogName)
> > > > > > > > > > > > > method
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > remove the instance corresponding to
> the
> > > > > original
> > > > > > > > > > > catalogName
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > necessary.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it will confuse users that
> > > > > > > > > > > `registerCatalog(String
> > > > > > > > > > > > > > > > > > > catalogName,Catalog catalog)` in the
> > > > > `Map<String,
> > > > > > > > > > Catalog>
> > > > > > > > > > > > > > > catalogs`
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > `registerCatalog(String catalogName,
> > > > > Map<String,
> > > > > > > > > String>
> > > > > > > > > > > > > > > properties)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This could potentially lead to
> > confusion. I
> > > > > > suggest
> > > > > > > > > > > changing
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > method name, perhaps to
> addCatalog(String
> > > > > > > > catalogName,
> > > > > > > > > > > > > > Map<String,
> > > > > > > > > > > > > > > > > > > String> properties), as previously
> > > mentioned
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Hang
> > > > > > > > > > > > > > > > > > > > add `registerCatalog(String
> > > > > catalogName,Catalog
> > > > > > > > > > catalog,
> > > > > > > > > > > > > > > > > > > boolean lazyInit)` method
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Since a catalog is already an instance,
> > > > adding
> > > > > > the
> > > > > > > > > > > "lazyInit"
> > > > > > > > > > > > > > > > > > > parameter to the registerCatalog(String
> > > > > > > catalogName,
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > catalog)
> > > > > > > > > > > > > > > > > > > method may not necessarily result in
> lazy
> > > > > > > > > initialization.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Do we need to think about encryption
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think encryption is necessary, but
> > > perhaps
> > > > > the
> > > > > > > > > > encryption
> > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > should be implemented in the
> CatalogStore
> > > > when
> > > > > > the
> > > > > > > > data
> > > > > > > > > > is
> > > > > > > > > > > > > > actually
> > > > > > > > > > > > > > > > > > > saved. For instance, the
> FileCatalogStore
> > > > could
> > > > > > > > encrypt
> > > > > > > > > > the
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > it is saved to a file, while the
> > > > > > MemoryCatalogStore
> > > > > > > > > does
> > > > > > > > > > > not
> > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > encryption.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Do we really need the
> > > > `MemoryCatalogStore`?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it is necessary to have a
> > > > > > > MemoryCatalogStore
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > > > default
> > > > > > > > > > > > > > > > > > > implementation, which saves Catalog
> > > > > > configurations
> > > > > > > in
> > > > > > > > > > > memory.
> > > > > > > > > > > > > > > > > > > Otherwise, if we want to implement
> > > > asynchronous
> > > > > > > > loading
> > > > > > > > > > of
> > > > > > > > > > > > > > Catalog,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > would need to introduce additional
> cache.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > @Xianxun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > What if asynchronous registration
> > failed?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This is also a critical concern. When
> > > > > executing
> > > > > > > DDL,
> > > > > > > > > > DQL,
> > > > > > > > > > > or
> > > > > > > > > > > > > DML
> > > > > > > > > > > > > > > > > > > statements that reference a specific
> > > Catalog,
> > > > > the
> > > > > > > > > Catalog
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > > > > will be initialized if it has not been
> > > > > > initialized
> > > > > > > > yet.
> > > > > > > > > > If
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > initialization process fails, these
> > > > statements
> > > > > > will
> > > > > > > > not
> > > > > > > > > > be
> > > > > > > > > > > > > > executed
> > > > > > > > > > > > > > > > > > > successfully.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > nt. The Map<String, Catalog> catalogs
> > and
> > > > > > > > > CatalogStore
> > > > > > > > > > > > > > > catalogstore
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the CatalogManager all have the same
> > > catalog
> > > > > > name,
> > > > > > > > but
> > > > > > > > > > > > > > correponding
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > different catalog instances.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > This issue can be resolved by referring
> > to
> > > > the
> > > > > > > > previous
> > > > > > > > > > > > > > responses.
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > key principle is that the Catalog that
> > has
> > > > been
> > > > > > > most
> > > > > > > > > > > recently
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > > will be given priority for subsequent
> > use.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Sun, Apr 2, 2023 at 10:58 PM Xianxun
> > Ye
> > > <
> > > > > > > > > > > > > > > yesorno828423@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Feng,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for driving this Flip, I do
> > > believe
> > > > > this
> > > > > > > > Flip
> > > > > > > > > > > could
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > helpful
> > > > > > > > > > > > > > > > > > > for users. The firm I work for also
> > > manages a
> > > > > lot
> > > > > > > of
> > > > > > > > > > > > catalogs,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > submission of tasks becomes slow
> because
> > of
> > > > > > > loading a
> > > > > > > > > > > number
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > catalogs.
> > > > > > > > > > > > > > > > > > > We obtain the catalogs in the user's
> > Flink
> > > > SQL
> > > > > > > > through
> > > > > > > > > > > > regular
> > > > > > > > > > > > > > > > > expression
> > > > > > > > > > > > > > > > > > > to avoid loading all catalogs to
> improve
> > > the
> > > > > > speed
> > > > > > > of
> > > > > > > > > > task
> > > > > > > > > > > > > > > > submission.
> > > > > > > > > > > > > > > > > > > After reading flip-295, I have some
> > > > questions:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. When creating a catalog with
> CREATE
> > > > > CATALOG,
> > > > > > > the
> > > > > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > > > registration method is used by default.
> > > What
> > > > if
> > > > > > > > > > > asynchronous
> > > > > > > > > > > > > > > > > registration
> > > > > > > > > > > > > > > > > > > failed? And how to implement
> asynchronous
> > > > > > > > registration?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2. I also have the same question with
> > > > > Shammon’s
> > > > > > > > first
> > > > > > > > > > > > > comment.
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > Map<String, Catalog> catalogs and
> > > > CatalogStore
> > > > > > > > > > catalogstore
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > CatalogManager all have the same
> catalog
> > > > name,
> > > > > > but
> > > > > > > > > > > > correponding
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > different catalog instances. For
> example:
> > > > > > > catalog1->
> > > > > > > > > > > > > jdbcCatalog
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > catalogs, but catalog1 -> hiveCatalog
> in
> > > the
> > > > > > > > > > catalogstore.
> > > > > > > > > > > > > Would
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > happen? Maybe we should define
> `catalogs`
> > > and
> > > > > > > > > > > `catalogstore`
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > clearly.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Xianxun
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2023年3月31日 11:13，Hang Ruan <
> > > > > > > > ruanhang1993@gmail.com
> > > > > > > > > >
> > > > > > > > > > > 写道：
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > MemoryCatalogStore
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>